Add metrics for messages to and from clients

[alexbozhenko]

Existing In/Out Msgs/Bytes metrics account for traffic from clients, routes, leafnodes and gateways.
So it is hard to see how actual offered load and what is sent to clients changes over time.

Add new metrics that only counts messages from and to clients.

On this Surveyor dashboard screenshot, I was publishing to s1 and subscribing on s3.
Old and new graphs show different picture:

And here is an example of values in /varz:

curl -Ss http://localhost:8222/varz http://localhost:8223/varz http://localhost:8224/varz | jq    '{server: .server_name,  in_msgs: .in_msgs, in_bytes: .in_bytes, in_client_msgs: .in_client_msgs,  in_client_bytes: .in_client_bytes, out_msgs: .out_msgs,    out_bytes: .out_bytes, out_client_msgs: .out_client_msgs, out_client_bytes: .out_client_bytes }'
{
  "server": "s1",
  "in_msgs": 10000152,
  "in_bytes": 1280031039,
  "in_client_msgs": 10000000,
  "in_client_bytes": 1280000000,
  "out_msgs": 8451913,
  "out_bytes": 1081861183,
  "out_client_msgs": 0,
  "out_client_bytes": 0
}
{
  "server": "s2",
  "in_msgs": 107,
  "in_bytes": 33648,
  "in_client_msgs": 0,
  "in_client_bytes": 0,
  "out_msgs": 98,
  "out_bytes": 29726,
  "out_client_msgs": 0,
  "out_client_bytes": 0
}
{
  "server": "s3",
  "in_msgs": 8451882,
  "in_bytes": 1081857677,
  "in_client_msgs": 0,
  "in_client_bytes": 0,
  "out_msgs": 8451885,
  "out_bytes": 1081856095,
  "out_client_msgs": 8451755,
  "out_client_bytes": 1081824640
}

Signed-off-by: Alex Bozhenko alexbozhenko@gmail.com

[MauriceVanVeen]

Should there also be outClientMsgs/Bytes?

[MauriceVanVeen]

LGTM

[bruth]

I am not opposed to having the standalone client counter, but I had noted before that you can calculate the client sent/recv by subtracting the total from the route, gateway, and leaf metrics that are already present. For reference, this was introduced here: #6967 and a follow-up here #7300.