Apps network usage metrics missing

[michaelgrifalconi]

Issue

Follow up of this slack discussion

Unlike CPU, memory and disk, there is no metric available with information about network (how much data is being sent/received by the container).

Context

Silk release allows to limit the amount of network bandwidth an application can use from the host (diego-cell) but there is no way to monitor if some applications are reaching the limit and being throttled or not.

Regardless of the limit being in place or not, I think knowing how much bandwidth is used by each application is useful and should be part of the already existing container metrics.

Steps to Reproduce

Look at data available in ContainerMetric

Expected result / Possible Fix

I would expect to see information about the network. Either a counter of sent / received bytes or an average bandwidth usage for the last period.

There was an implementation of the first option on the Garden API but was abandoned due to the swith to cf-networking (see slack)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/164866818

The labels on this github issue will be updated when the story is started.

[barakyo]

Hi @michaelgrifalconi,

We took a look into the metrics that Silk currently provides, which we found in poller.go of netmon. However, these are container to container traffic metrics and not external. In addition, they're only provided per diego cell and not an application basis. Maybe this can help you for now, until this story is prioritized.

We've updated the tracker story and included some acceptance criteria. Would you mind reviewing them to let us know if we forgot anything?

Acceptance Criteria

As an operator
I want to be able to view network metrics
So that I can monitor if some applications are reaching the limit and being throttled or not.

Given a deployment of Cloud Foundry
When I look at Explore DataDog metrics for tx_bytes
Then I see graphs for each running application
And when I send traffic out of the application
Then I see it increase in DataDog

Given a deployment of Cloud Foundry
When I look at Explore DataDog metrics for rx_bytes
Then I see graphs for each running application
And when I send traffic into the application
Then I see it increase in DataDog

Given a deployment of Cloud Foundry with a ASGs that restrict outgoing traffic
When I look at Explore DataDog metrics for tx_dropped
Then I see graphs for each running application
And when I send traffic out of the application
Then I see it increase in DataDog

Given a deployment of Cloud Foundry with throttling
When I look at Explore DataDog metrics for rx_dropped
Then I see graphs for each running application
And when I send traffic into the application
Then I see it increase in DataDog

Thanks!
@ameowlia and @barakyo, CF Networking Program Members

[michaelgrifalconi]

Thank you for creating the story! I think it looks fine and describes well the topic.

Best,
Michael

[andrew-edgar]

We would be very interested in this sort of thing as well. We want to be able to know which apps/instances are showing high network usage across the whole env. not only to limit but also to have the ability to see if some apps are using more than they should.

The limits on bandwidth (rate, burst) are also very heavy handed as they apply to ALL instances system wide. where perhaps it would be good to be able to limit things more by org/space

[ionphractal]

I'm also looking for this (cloudfoundry/silk-release#16) and I would like to add here that some IaaS layer providers may charge traffic volume and these statistics would be vital for sharing costs amongst app owners. Maybe it would be good to have this as (or inside) an app usage event as well, so that it can be directly integrated into present billing mechanisms.

[mcwumbly]

Hey folks, we're adding the help wanted label on this issue as it is something we think makes sense but doesn't fit into our near team priorities. if anyone is interested in picking this up and wants to chat about implementation ideas, please share your thoughts here or find us in the #networking channel on CloudFoundry Slack.

[mariash]

Silk provides metrics about data sent by containers. This would be iptables metrics which configure security groups for containers controlling outgoing traffic and incoming traffic from other containers. The burst and rate limit configuration options you mentioned control outgoing traffic only. All other incoming traffic is not controlled by silk. Please let us know if these metrics are enough?

[mkocher]

@mariash Are those container metrics like cpu/memory/disk/log-rate? I think the ones you're linking to are system component metrics.

[mariash]

@mkocher you are right this is not for each container. We use this for the whole silk-vtep interface. I guess this package can be used to obtain data for each container interface.