Support dropping of specific attributes from instruments

[dpk83]

What are you trying to achieve?
Use Views to customize the metrics output to drop specific attributes from metrics when present.

Currently the SDK provides the customization as follows

In this same scenario we have additional mechanism to let users add additional tags. Not all the tags are known at the time of creation of the instrument.

Our scenario is that we want to drop certain tags (e.g. HTTP verb in the above example) while keeping all other tags i.e. the tags that were added to the original instrument as well as the tags that were used to enrich the original instruments.

Currently OpenTelemetry provides

A list of attribute keys (optional). If provided, the attributes that are not in the list will be ignored. If not provided, all the attribute keys will be used by default

This doesn't work for our scenarios because this only let you keep the tags that are specified. In our scenarios users don't always know all the tags which will be added to the instrument.

What did you expect to see?

A list of attribute keys (optional). If provided, the attributes that are in the list will be ignored.

Additional context.

[dpk83]

@reyang

[pirgeo]

Some thoughts:
I am not against this change, but it surprises me that you have a situation where you don't know your metric attribute keys beforehand. This sounds to me like a dangerous situation, leading to a potential cardinality explosion when adding new metrics streams that you don't know about at first.

I do think that there could be situations where dropping a few attributes is more user-friendly than having to specify all of them in the first place, but it also raises questions of which to apply first.

[dpk83]

You are thinking simple small services. In complex services and complex organizations where multiple services are using common libraries and there are different places in the code that needs to add additional metadata to the requests. There are some which are quite visible to the user and some not quite. It's a lot of context to write here for the full set of scenarios so I will just stay with the gist.

Another point is everything needs to be configurable for our services, so as a services owner I should be able to decide to drop certain attributes quickly (if it turns out they are causing some problems) without needing to modify the code and having to wait to go through the full deployment cycle which is generally not an option when incidents hit.

Take this scenario: There is an incident caused by some attribute in a metric which is high cardinality or ended up having some sensitive information (reason could be anything, human mistakes, bugs etc). As part of handling the incident I immediately need to stop that dimension from being emitted. At this point I don't expect the developer to scramble to find out all the attributes for that specific metric and then list down all of them but one in the config. The expectation here would be to just drop that single attribute via config quickly.

[pirgeo]

I see, that makes sense to me. I was going to point out that you likely have some sort of dashboard that shows you all the metrics/attributes you have collected, so there you already have some information about the attribute keys you ingested, but as you said, it is easy to miss something that was not yet emitted or similar.

Since you mentioned having to go through a deployment cycle - I am not aware of any mechanism for configuring Views without code changes. I think you will have to go through the cycle either way. One possible way (although I don't think this is standardized in the spec) is using AttributeProcessors (or AttributeFilters). See Java for example: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/ViewBuilder.java#L72. This would give you a way of implementing your own logic to filter attributes with a config that changes at runtime.

I think you are not looking for a quick fix, but I am pretty sure a collector would also give you what you want here. There are other considerations there (data locality, deploying an extra service, etc.) that I am sure you are aware of. The collector has an attributes processor that allows you to delete certain attributes (although I think changing configuration currently also requires a restart of the collector): https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor

[dpk83]

Since you mentioned having to go through a deployment cycle - I am not aware of any mechanism for configuring Views without code changes.

We have a framework built on top of OpenTelemetry to make most of the things configurable for our services. Ability to control behavior via configuration is very essential for us.

[cijothomas]

Can be closed with #4188