[FEATURE REQUEST]: As a CF Operator, I expect that when I observe the "AppInstanceExceededLogRateLimitCount" metric, I can see the app instance details that caused this value to be incremented so I can take action with the app owner

[sunjayBhatia]

Is your feature request related to a problem?

Right now, when AppInstanceExceededLogRateLimitCount is emitted (here), it is emitted as a per-cell tagged counter metric. The loggregator agent adds just the cell-specific tags to the metric, nothing more, resulting in metrics that look like this:

deployment:"cf" job:"diego-cell" index:"0e98fd00-47b2-4589-94f0-385f78b3a04d" ip:"10.0.1.12" tags:<key:"instance_id" value:"0e98fd00-47b2-4589-94f0-385f78b3a04d" > tags:<key:"source_id" value:"rep" > counterEvent:<name:"AppInstanceExceededLogRateLimitCount" delta:1 total:206 >

As a result, operators must do some additional work to filter the app instances on the cell this metric comes from to identify the actual culprit of the chatty logging. This is not super straightforward and we can do better for operators so that it is much easier to identify problematic apps (and even app instances).

Describe the solution you'd like

In the the log rate limit reporter, we have access to the metric tags set on the app's desired LRP and use them to push our own log line into the app log stream. We should be able to do the same thing with our AppInstanceExceededLogRateLimitCount metric in order to tag it accordingly and ensure the value emitted is not a per-cell metric but a per-app instance metric.

We can potentially just make a new version of our IncrementCounter method to add tags to the envelope we want to send using this option.

Diego repo

executor

Describe alternatives you've considered

• An alternative is to do the emission logic here instead keeping in the log streamer package. This package already has reporters that periodically emit per-container (app instance) metrics.
  • maybe implement another metric reporter that is available to the log streamer
  • IMO, moving the actual call to the loggregator/metron client out to a goroutine separate from the main functionality is "safer," the way this code is currently written if the call to loggregator blocks, the work we care about is blocked; we use the parallel goroutine for metric emission pattern a lot I believe for this sort of purpose, and to keep metric emission logic/periodicity in one place so we don't have to re-implement periodic metrics repeatedly
• If the "counter" type metric does not work, we can try to use a gauge style metric instead

[mdelillo]

In lieu of emitting app instance details in the log message, we have come up with some cf commands that will show you that data using existing plugins.

First, install the firehose and log-cache plugins:

cf install-plugin 'Firehose Plugin'
cf install-plugin 'log-cache'

This command will show when any app has exceeded the log rate: cf tail -f rep | grep AppInstanceExceededLogRateLimitCount

And this command will show the specific app which has exceeded the rate: cf nozzle -f LogMessage | grep 'app instance exceeded log rate limit'

[sunjayBhatia]

In lieu of emitting app instance details in the log message

I am not suggesting emitting app instance details in the log message, but instead tagging the counter metric (here) we emit with the same tags as the log message we put in the app log stream so someone aggregating metrics can drill down into the metric values for individual apps. This would change the source/instance id of the metric to be from the app instance instead of the cell

I'm also curious how this cf nozzle approach will work on a larger foundation, inspecting each log line from every app to find the chatty ones seems expensive

[sunjayBhatia]

reopening so we can continue to chat about this

[robertjsullivan]

If we add a tag that adds application_instance_id and application_guid, the AppInstanceExceededLogRateLimitCount metric will have an enormous cardinality. This will create a large storage burden on downstream monitoring systems. Given that we expect operators will want to rectify the bad actors quickly, we would be storing a lot of metrics that are not being used.

I think that we should have the count be by diego cell only. Logging the offending app names to Diego's logs on a set cadence seems like a reasonable solution for identifying the bad actors.

[jvshahid]

As a CF Operator, I expect that when I observe the "AppInstanceExceededLogRateLimitCount" metric,

@sunjayBhatia I don't understand why the operator is the persona of interest in this case. I thought the whole purpose of this story was to avoid having applications negatively affect their neighbors. I think the app developer will be more interested in knowing that their application logs are getting dropped.

If we add a tag that adds application_instance_id and application_guid, the AppInstanceExceededLogRateLimitCount metric will have an enormous cardinality.

@robertjsullivan How is this different from the CPU and Memory metrics that we are currently emitting?

[aminjam]

@robertjsullivan How is this different from the CPU and Memory metrics that we are currently emitting?

for CPU and memory, we end up emitting the application_id as the source_id (https://github.com/cloudfoundry/diego-logging-client/blob/88015a7/client.go#L217), as opposed to keeping the source_id as "rep", but emitting application level information under that metric

[sunjayBhatia]

I would think we would also want to change the source id to be the application id to make the metric useful to app developers if we were going to tag it as suggested

@jvshahid yeah, that is a good point, this could be reframed as an app developer facing feature (so they can keep track of their log usage, sort of another resource usage metric to key off of), or maybe we can conclude that app developers have enough info for now with the log we push into the log stream telling them their app has been rate limited

[robertjsullivan]

The cardinality isn't different than the cpu metric and memory metrics but I think those are more valuable as that data is constantly relevant. In this case, it's a counter that we'd be emitting for every instance, but only relevant to a small few.

If we wanted to emit a logging metric for applications, I think we should emit the total count of log lines from a container. An operator could use the rate() function to determine if it's violating the set threshold. App Devs would also find that useful as well.

[aminjam]

I'm also curious how this cf nozzle approach will work on a larger foundation, inspecting each log line from every app to find the chatty ones seems expensive

@sunjayBhatia We tested this in a large foundation and were able to detect the apps logging in as an admin.

Closing this issue since we have enough information for app developers to know their app is being rate limited.

Please feel free to re-open this if you'd like to see a different outcome.