Leverage ratelimit to ban IPs permanently

[santinoncs]

Hi,

I am using ratelimit envoyproxy service in conjunction with istio to perform the global rate-limit service in the istio-ingressgateway pods that are in front of my infrastructure inside a kubernetes cluster.
My goal is to be able to get the IPs that are being rate limited and store them in prometheus and use some process that
can read from prom and do some logic, like ban those IPs permanently via a cloud service like Google cloud Armor or one alike.

I am using this action in my "rate_limits filter"

    rate_limits:

      - actions:
        - request_headers:
            descriptor_key: remote_address_second
            header_name: x-envoy-external-address
        - destination_cluster: {}

So this is what in redis store looks like

entrypoint-v1-entrypoint_remote_address_second_188.2.75.xx_destination_cluster_outbound|80||$this_is_the_destination_cluster_1627892892

But with statsd-prom-exporter activated in prometheus I am able to see just this ( there is no IP ) so I can't have all the dimensions in the metric

ratelimit.service.rate_limit.entrypoint-v1-entrypoint.remote_address_minute.destination_cluster_outbound|80||$this_is_the_destination_cluster.over_limit: 62

Is there any way I can "see" in statsd ( :6070/stats ) the rate limit actions in the metric? being in this case the IP ( header x-envoy-external-address )

Is there any modification needed in this code to achieve that?

[ysawa0]

so the "cleanest" way to do this is put some logic in the go code to track IPs ratelimited and periodically push the data to somewhere like s3.
but to answer your question: I believe that data is being collected in stats. if you don't see it in prom, you may need to add a new rule in the prom statsd config to parse that particular metric since the existing config https://github.com/envoyproxy/ratelimit/blob/main/examples/prom-statsd-exporter/conf.yaml covers only minimal amount of cases.

[santinoncs]

Thanks for your response!

I cannot see these metrics in statsd ( :6070/stats ), so i assume that there are not in prometheus too, even modifying the conf.yaml.

# curl localhost:6070/stats
...
ratelimit.service.rate_limit.serviceA.remote_address_minute.destination_cluster_outbound|80|prod|hostname_serviceA.near_limit: 0
..
ratelimit.service.rate_limit.serviceA.remote_address_minute.destination_cluster_outbound|80||hostname_serviceB.near_limit: 0
...
ratelimit.service.rate_limit.serviceB.remote_address_minute.destination_cluster_outbound|80||hostname_serviceB.total_hits: 131495
...

Is this assumption correct?

[ysawa0]

Gotcha. Can you try pulling in #283 to your testing and checking if it shows up in /metrics on prom. this is what I see in /metrics when I run the examples.

# HELP ratelimit_service_rate_limit_near_limit Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_near_limit counter
ratelimit_service_rate_limit_near_limit{domain="rl",key1="foo",key2="bar"} 2
# HELP ratelimit_service_rate_limit_over_limit Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_over_limit counter
ratelimit_service_rate_limit_over_limit{domain="rl",key1="foo",key2="bar"} 85

I'm not that familiar with statsd so I could be wrong but: I think there was a regression somewhere in the stats code and viewing of metrics has been wonky since.

[santinoncs]

thanks ! let me try

[ysawa0]

I didn't do a 100% deep dive but here's what I think happened. We upgraded gostats to 0.4.0 in #141 which removed publishing of stats via expvars: https://github.com/lyft/gostats/releases/tag/v0.3.0
Since /stats endpoint relied on expvars https://github.com/envoyproxy/ratelimit/blob/main/src/server/server_impl.go#L252, it now doesn't display any ratelimit stats except some default memstats/cmdline stuff.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.