Heartbeat event format #4091

urso · 2017-04-23T17:38:27Z

PR addresses #3406 and #3838.

Additional work on extracting meta-data from connection layers should be done in follow-up PR.

Work done:

add look.Status
update event format per monitor
Rewrite heartbeat dialchain package:
- simplify package
- have connection layers add standardized connection data to the final event
- add some more helpers
- add some godocs

Event format changes:

Old Field	Changes
monitor	-> monitor.id
type	-> monitor.name defaults to monitor.type if not configured introduce new `type` fields with default 'monitor' until `type` is not required anymore
duration	-> monitor.duration
scheme	-> monitor.scheme
up	-> monitor.status - replace boolean with keywords 'up', 'down'
host	-> monitor.host -> resolve.host
ip	-> monitor.ip -> resolve.ip
resolve_rtt	-> resolve.rtt
icmp_rtt	-> icmp.rtt
port	-> tcp.port
tcp_connect_rtt	-> tcp.rtt.connect
validate_rtt	-> tcp.rtt.validate
socks5_connect_rtt	-> socks5.rtt.connect
tls_handshake_rtt	-> tls.rtt.handshake
url	-> http.url
response.status	-> http.response.status
rtt	-> http.rtt.total
http_rtt	-> http.rtt.response_header

New fields:

type: always set to "monitor". Can be removed when not required by libbeat anymore
monitor.type: the monitor type, e.g. "tcp", "icmp", "http".
http.rtt.validate: Duration between first byte of HTTP request being written and response being processed by validator. Duration based on already available network connection.
http.rtt.validate_body: Duration of validator required to read and validate the response
http.rtt.write_request: Duration of writing the complete HTTP request. Duration based on already available network connection.

Sample Events:

ICMP Event:

{
  "@timestamp": "2017-04-24T11:36:53.639Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "icmp": {
    "requests": 1,
    "rtt": {
      "us": 87
    }
  },
  "monitor": {
    "duration": {
      "us": 2557
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "myname",
    "status": "up",
    "type": "icmp"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 2316
    }
  }
}

ICMP event for unknown host:

{
  "@timestamp": "2017-04-24T11:36:53.639Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "error": {
    "message": "lookup unknown-host: no such host",
    "type": "io"
  },
  "monitor": {
    "duration": {
      "us": 7628
    },
    "host": "unknown-host",
    "name": "myname",
    "status": "down",
    "type": "icmp"
  },
  "resolve": {
    "host": "unknown-host"
  }
}

TCP Event:

{
  "@timestamp": "2017-04-24T12:29:04.649Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "monitor": {
    "duration": {
      "us": 4461
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "tcp",
    "scheme": "tcp", // Note: can be 'tcp' or 'tls' for TCP monitor. HTTP monitor sets 'http' or 'https'
    "status": "up",
    "type": "tcp"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 4136
    }
  },
  "tcp": {
    "port": "6060",
    "rtt": {
      "connect": {
        "us": 183
      }
    }
  }
}

TCP + TLS Event:

{
  "@timestamp": "2017-04-24T12:29:04.649Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "monitor": {
    "duration": {
      "us": 234591
    },
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "name": "tcp",
    "scheme": "tls",
    "status": "up",
    "type": "tcp"
  },
  "resolve": {
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "rtt": {
      "us": 51663
    }
  },
  "tcp": {
    "port": "443",
    "rtt": {
      "connect": {
        "us": 46140
      }
    }
  },
  "tls": {
    "rtt": {
      "handshake": {
        "us": 136605
      }
    }
  }
}

HTTP Event with 404:

{
  "@timestamp": "2017-04-24T13:44:44.718Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "error": {
    "message": "404 Not Found",
    "type": "validate"
  },
  "http": {
    "response": {
      "status": 404
    },
    "rtt": {
      "content": {
        "us": 244
      },
      "response_header": {
        "us": 225
      },
      "total": {
        "us": 537
      },
      "write_request": {
        "us": 63
      }
    },
    "url": "http://localhost:6060"
  },
  "monitor": {
    "duration": {
      "us": 4935
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "http",
    "scheme": "http",
    "status": "down",
    "type": "http"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 4271
    }
  },
  "tcp": {
    "port": "6060",
    "rtt": {
      "connect": {
        "us": 247
      }
    }
  }
}

HTTPS event:

{
  "@timestamp": "2017-04-24T13:44:44.718Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "http": {
    "response": {
      "status": 200
    },
    "rtt": {
      "content": {
        "us": 167523
      },
      "response_header": {
        "us": 167433
      },
      "total": {
        "us": 339594
      },
      "write_request": {
        "us": 102
      }
    },
    "url": "https://play.golang.org"
  },
  "monitor": {
    "duration": {
      "us": 344043
    },
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "name": "http",
    "scheme": "https",
    "status": "up",
    "type": "http"
  },
  "resolve": {
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "rtt": {
      "us": 4272
    }
  },
  "tcp": {
    "port": "443",
    "rtt": {
      "connect": {
        "us": 38350
      }
    }
  },
  "tls": {
    "rtt": {
      "handshake": {
        "us": 133618
      }
    }
  }
}

elasticmachine · 2017-04-23T18:10:10Z

heartbeat/monitors/active/dialchain/dialchain.go

-
-func UDPDialer(name string, to time.Duration) NetDialer {
-	return NetDialer{name, transport.NetDialer(to)}
+func (c *DialerChain) TestBuild() error {


[golint] _{reported by reviewdog 🐶}
exported method DialerChain.TestBuild should have comment or be unexported

elasticmachine · 2017-04-24T14:27:40Z

heartbeat/monitors/util.go

@@ -53,16 +52,16 @@ func (s IPSettings) Network() string {
 	return ""
 }

-func MakeSimpleJob(name, typ string, f func() (common.MapStr, error)) Job {
-	return MakeJob(name, typ, func() (common.MapStr, []TaskRunner, error) {
+func MakeSimpleJob(jobName string, f func() (common.MapStr, error)) Job {


[golint] _{reported by reviewdog 🐶}
exported function MakeSimpleJob should have comment or be unexported

elasticmachine · 2017-04-24T14:27:40Z

heartbeat/monitors/util.go

-func MakeJob(name, typ string, f func() (common.MapStr, []TaskRunner, error)) Job {
-	return &funcJob{name, typ, funcTask{func() (common.MapStr, []TaskRunner, error) {
-		return annotated(time.Now(), typ, f).Run()
+func MakeJob(jobName string, f func() (common.MapStr, []TaskRunner, error)) Job {


[golint] _{reported by reviewdog 🐶}
exported function MakeJob should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

 type IPSettings struct {
 	IPv4 bool     `config:"ipv4"`
 	IPv6 bool     `config:"ipv6"`
 	Mode PingMode `config:"mode"`
 }

+type JobSettings struct {


[golint] _{reported by reviewdog 🐶}
exported type JobSettings should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

+	Fields common.MapStr
+}
+
+type HostJobSettings struct {


[golint] _{reported by reviewdog 🐶}
exported type HostJobSettings should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

@@ -342,3 +444,42 @@ func filterIPs(ips []net.IP, filt func(net.IP) bool) []net.IP {
 	}
 	return out
 }
+
+func MakeJobSetting(name string) JobSettings {


[golint] _{reported by reviewdog 🐶}
exported function MakeJobSetting should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

+	return JobSettings{Name: name}
+}
+
+func (s JobSettings) WithFields(m common.MapStr) JobSettings {


[golint] _{reported by reviewdog 🐶}
exported method JobSettings.WithFields should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

+	return s
+}
+
+func (s *JobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }


[golint] _{reported by reviewdog 🐶}
exported method JobSettings.AddFields should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

+
+func (s *JobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }
+
+func MakeHostJobSettings(name, host string, ip IPSettings) HostJobSettings {


[golint] _{reported by reviewdog 🐶}
exported function MakeHostJobSettings should have comment or be unexported

elasticmachine · 2017-04-26T12:42:15Z

heartbeat/monitors/util.go

+	return HostJobSettings{Name: name, Host: host, IP: ip}
+}
+
+func (s HostJobSettings) WithFields(m common.MapStr) HostJobSettings {


[golint] _{reported by reviewdog 🐶}
exported method HostJobSettings.WithFields should have comment or be unexported

elasticmachine · 2017-04-26T12:42:16Z

heartbeat/monitors/util.go

+	return s
+}
+
+func (s *HostJobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }


[golint] _{reported by reviewdog 🐶}
exported method HostJobSettings.AddFields should have comment or be unexported

ruflin

Nice cleanup. It makes the dialer part much easier to follow. As this is a major change on how it works I'm happy to merge it after the minor comments are adressed and then to further iterations from there.

ruflin · 2017-04-26T13:50:05Z

heartbeat/_meta/fields.yml

+
+            - name: validate_body
+              type: group
+              description: |


Normally we use a >. I assume | also works?

oh, oversight from my side. in YAML one can use >, >- or |.

>: folded-style. Replace newlines with space. Multiple blank lines are replaced with newline (or space in some implementations)
|: literal block. Keeps newlines and blank lines.

I wanted to enforce the newline (or better blank line?) before "Note:" in the hopes of the asciidoc putting the note in a separate new line. That's why I used |.

If I remember correctly >- replaces a blank line (\n\n) by one newline (\n).

ruflin · 2017-04-26T13:56:11Z

heartbeat/monitors/active/http/task.go

@@ -9,6 +9,7 @@ import (
 	"net/http"
 	"strconv"
 	"strings"
+


empty line too much?

interesting. so gimport woes here? Will fix.

ruflin · 2017-04-26T13:57:08Z

heartbeat/monitors/active/http/task.go

@@ -41,25 +42,25 @@ func newHTTPMonitorHostJob(
 		return nil, err
 	}

-	hostname, port, err := splitHostnamePort(request)
+	hostname, _, err := splitHostnamePort(request)


port is excluded because it is already in the tcp layer, right?

Port is missing here. This constructor is used when a proxy is configured + name lookup happens by proxy itself. I did undo the tcp dialer to report the port number due to potential use of proxies.

ruflin · 2017-04-26T14:02:24Z

heartbeat/monitors/util.go

-		case 1:
-			return makeTask(cont[0])
+		case
+			1:


is the 1: intentional on a new line?

ruflin · 2017-04-26T14:03:37Z

heartbeat/monitors/util.go

 	return MakeCont(func() (common.MapStr, []TaskRunner, error) {
+		fmt.Println("run task")


ruflin · 2017-04-26T14:03:43Z

heartbeat/monitors/util.go

 		event, cont, err := fn()
+		fmt.Println("task returns: ", event, cont, err)


ruflin · 2017-04-26T14:03:52Z

heartbeat/monitors/util.go

 		if err != nil {
 			if event == nil {
 				event = common.MapStr{}
 			}
+			fmt.Println("set error: ", err)


ruflin · 2017-04-26T14:03:58Z

heartbeat/monitors/util.go

 			event["error"] = look.Reason(err)
 		}

 		if event != nil {
-			event.Update(common.MapStr{
+			status := look.Status(err)
+			fmt.Println("set status: ", status)


ruflin

Could you rebase and squash?

See PR elastic#4091 for detailed list of event format changes and sample events. - export transport.DialWith - update heartbeat look: - add look.Status - add some missing godoc - rewrite heartbeat dialchain package - simplify package - have connection layers add standardized connection data to the final event - add some more helpers - add some godocs - update event format. See PR elastic#4091 - add more detailed duration measure to HTTP module - update job settings - introduce explicit structs for job settings - job settings can contain static monitor fields to be added to every event - helpers can add more static monitor fields to settings - update fields.yml structure - update kibana dashboard

urso added Heartbeat in progress Pull request is currently in progress. labels Apr 23, 2017

elasticmachine reviewed Apr 23, 2017

View reviewed changes

urso force-pushed the heartbeat/event-format branch from f1b1ea0 to 9f2f09d Compare April 24, 2017 10:14

ruflin mentioned this pull request Apr 24, 2017

[WIP] Update heartbeat event structure #4054

Closed

elasticmachine reviewed Apr 24, 2017

View reviewed changes

elasticmachine reviewed Apr 26, 2017

View reviewed changes

urso changed the title ~~[WIP] Heartbeat event format~~ Heartbeat event format Apr 26, 2017

urso added review and removed in progress Pull request is currently in progress. labels Apr 26, 2017

ruflin reviewed Apr 26, 2017

View reviewed changes

ruflin approved these changes Apr 27, 2017

View reviewed changes

urso force-pushed the heartbeat/event-format branch from c0ead6e to 2b7cff7 Compare April 27, 2017 14:57

urso force-pushed the heartbeat/event-format branch from 2b7cff7 to 6b4ee71 Compare April 27, 2017 15:05

ruflin merged commit 6b01fa9 into elastic:master Apr 28, 2017

This was referenced May 18, 2017

Heartbeat: Exported Fields - Name #3838

Closed

Heartbeat event data structure #3406

Closed

Heartbeat rtt.us field is not documented #3530

Closed

urso mentioned this pull request Nov 23, 2017

Heartbeat ports exported as string #5696

Closed

urso deleted the heartbeat/event-format branch February 19, 2019 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heartbeat event format #4091

Heartbeat event format #4091

urso commented Apr 23, 2017 •

edited

Loading

elasticmachine Apr 23, 2017

elasticmachine Apr 24, 2017

elasticmachine Apr 24, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

elasticmachine Apr 26, 2017

ruflin left a comment

ruflin Apr 26, 2017

urso Apr 26, 2017

ruflin Apr 26, 2017

urso Apr 26, 2017

ruflin Apr 26, 2017

urso Apr 26, 2017

ruflin Apr 26, 2017

urso Apr 26, 2017

ruflin Apr 26, 2017

urso Apr 26, 2017

ruflin Apr 26, 2017

ruflin Apr 26, 2017

ruflin Apr 26, 2017

ruflin left a comment


		func (s *JobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }

		func MakeHostJobSettings(name, host string, ip IPSettings) HostJobSettings {

		return MakeCont(func() (common.MapStr, []TaskRunner, error) {
		fmt.Println("run task")

		event, cont, err := fn()
		fmt.Println("task returns: ", event, cont, err)

Heartbeat event format #4091

Heartbeat event format #4091

Conversation

urso commented Apr 23, 2017 • edited Loading

Event format changes:

Sample Events:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

urso commented Apr 23, 2017 •

edited

Loading