Skip to content

Make Alert Manager API Concurrency configurable #5412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 21, 2023
Merged

Make Alert Manager API Concurrency configurable #5412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
* [FEATURE] Query Frontend: Add `cortex_rejected_queries_total` metric for throttled queries. #5356
* [FEATURE] Querier: Log query stats when querying store gateway. #5376
* [FEATURE] Querier/StoreGateway: Allow the tenant shard sizes to be a percent of total instances. #5393
* [FEATURE] Added the flag `-alertmanager.api-concurrency` to configure alert manager api concurrency limit. #5412
* [ENHANCEMENT] Distributor/Ingester: Add span on push path #5319
* [ENHANCEMENT] Support object storage backends for runtime configuration file. #5292
* [ENHANCEMENT] Query Frontend: Reject subquery with too small step size. #5323
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration/config-file-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,10 @@ cluster:
# CLI flag: -experimental.alertmanager.enable-api
[enable_api: <boolean> | default = false]

# Maximum number of concurrent GET API requests before returning an error.
# CLI flag: -alertmanager.api-concurrency
[api_concurrency: <int> | default = 0]

alertmanager_client:
# Timeout for downstream alertmanagers.
# CLI flag: -alertmanager.alertmanager-client.remote-timeout
Expand Down
2 changes: 2 additions & 0 deletions pkg/alertmanager/alertmanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ type Config struct {
Replicator Replicator
Store alertstore.AlertStore
PersisterConfig PersisterConfig
APIConcurrency int
}

// An Alertmanager manages the alerts for one user.
Expand Down Expand Up @@ -266,6 +267,7 @@ func New(cfg *Config, reg *prometheus.Registry) (*Alertmanager, error) {
GroupFunc: func(f1 func(*dispatch.Route) bool, f2 func(*types.Alert, time.Time) bool) (dispatch.AlertGroups, map[model.Fingerprint][]string) {
return am.dispatcher.Groups(f1, f2)
},
Concurrency: am.cfg.APIConcurrency,
})
if err != nil {
return nil, fmt.Errorf("failed to create api: %v", err)
Expand Down
5 changes: 4 additions & 1 deletion pkg/alertmanager/multitenant.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ type MultitenantAlertmanagerConfig struct {

Cluster ClusterConfig `yaml:"cluster"`

EnableAPI bool `yaml:"enable_api"`
EnableAPI bool `yaml:"enable_api"`
APIConcurrency int `yaml:"api_concurrency"`

// For distributor.
AlertmanagerClient ClientConfig `yaml:"alertmanager_client"`
Expand Down Expand Up @@ -117,6 +118,7 @@ func (cfg *MultitenantAlertmanagerConfig) RegisterFlags(f *flag.FlagSet) {
f.DurationVar(&cfg.PollInterval, "alertmanager.configs.poll-interval", 15*time.Second, "How frequently to poll Cortex configs")

f.BoolVar(&cfg.EnableAPI, "experimental.alertmanager.enable-api", false, "Enable the experimental alertmanager config api.")
f.IntVar(&cfg.APIConcurrency, "alertmanager.api-concurrency", 0, "Maximum number of concurrent GET API requests before returning an error.")

f.BoolVar(&cfg.ShardingEnabled, "alertmanager.sharding-enabled", false, "Shard tenants across multiple alertmanager instances.")
f.Var(&cfg.EnabledTenants, "alertmanager.enabled-tenants", "Comma separated list of tenants whose alerts this alertmanager can process. If specified, only these tenants will be handled by alertmanager, otherwise this alertmanager can process alerts from all tenants.")
Expand Down Expand Up @@ -965,6 +967,7 @@ func (am *MultitenantAlertmanager) newAlertmanager(userID string, amConfig *amco
Store: am.store,
PersisterConfig: am.cfg.Persister,
Limits: am.limits,
APIConcurrency: am.cfg.APIConcurrency,
}, reg)
if err != nil {
return nil, fmt.Errorf("unable to start Alertmanager for user %v: %v", userID, err)
Expand Down