Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(global-filters): Introduce global generic filters (#3161)
This PR adds support for global generic filters. Generic filters are a new schema of inbound filters with generic conditions to avoid creating a new filter for each use case. Project configs already support [project-specific generic filters](ee3036c). Global generic filters have been designed to be extendable, future-compatible, independent, and flexible to work well in isolation but also with project generic filters. For event processing, Relay uses both the project config of the event and the global config. Before running the filters, Relay will merge the generic filters from the two configs and apply these to the event. If there's a match relay will drop the event and generate an outcome, or forward the event to the next processing step if there isn't. ## Why Currently, adding a new generic filter results in an increase in the size of project configs (`config_size * number_of_filters`). This can result in a significant increase in memory usage in all the services that store project configs (`increase_per_project * number_of_projects`), especially Relay. Incorporating generic filters into global configs allows Relay to apply these filters similarly with minimal overhead. There are additional, less important goals that this approach also achieves: not increasing network usage, not impacting project config computation time, etc. ## Previous work Generic inbound filters were introduced in project configs in ee3036c. They are currently not enabled on `sentry` as supporting these filters in the global config was prioritized first. ## Protocol Global generic filters have the same protocol as project generic filters, but they exist in the global config. A new optional `filters` key is added as a top-level key in global configs. This object expects the following properties: - `version`: the version of global configs, for future-compatibility, as a `u16`. - `filters`: a list of generic filters sorted by descending priority, with the same protocol as with the project generic filters. Example of a full protocol: ```json { // ... other global config keys ... "filters": { "version": 1, "filters": [ { "id": "sample-errors", "isEnabled": true, "condition": { "op": "eq", "name": "event.exceptions", "value": "Sample Error" } } ] } } ``` ## Decisions and trade-offs ### Protocol Supporting external Relays with future compatibility to make updates safely: - Introduced `version` to skip filters on a version not supported by Relay. - When filters are unsupported, Relay forwards events upstream without extracting metrics. This allows the next updated Relay in the chain supporting the filters to run filtering and, in case there's a decision to keep the event, extract metrics from it. Relay merges the configs prioritizing the project config. Each filter has an `isEnabled` flag to maximize flexibility and allow: - Defining a filter for a single project in its config. - Defining a filter for all projects in the global config. - Defining a filter for most projects by enabling it in the global config and disabling it in the applicable project configs. - Defining a filter for some projects by disabling it in the global config and enabling it in the applicable project configs. This is the simplest alternative to support all the use cases. ### Relay implementation details Relay types the filters internally as an ordered map with custom (de)serialization, keeping the order in the protocol, to benefit from: - `O(1)` retrieval of filters given the ID. - Removing duplicated filters. Removing duplicates is a prerequisite to merge the filters from both configs properly. Since the list of filters in the payload is expected to be sorted by descending priority, only the first occurrence of a filter is selected and the rest are discarded during deserialization. Discarded filters are not forwarded downstream. `filters` is typed as an `ErrorBoundary` instead of an `Option`: - Errors during deserialization indicate breaking changes. Relay skips metric extraction and dynamic sampling in these cases, and the next relay in chain deals with them. - Note: processing relays will run dynamic sampling and extract metrics even if filters are broken. - An empty entry in a payload is deserialized as an empty list of filters, effectively a noop. - A list of filters in a supported version is appropriately deserialized and applied.
- Loading branch information
1 parent
18a197e
commit 2800e95