This repository was archived by the owner on Jul 28, 2026. It is now read-only.
[dev.operator-integrations] Merge main into dev.operator-integrations - #1500
Merged
Conversation
* rfc-0001: add rules for when RFC PRs should be merged * use status field instead of merge to indicating state
* Parametrize logs daemonset K8s manifests * Update CHANGELOG.md
* Add depguard linter to reject packages we tend to avoid * Replace golint with revive, since golint is deprecated * Remove interfacer, which is deprecated with no replacement * Add makezero linter to detect misuse of make with append * Add tenv to prefer t.Setenv over os.Setenv in tests * Add whitespace to report unnecessary blank lines * Ignore test files for errcheck In addition to the above, the following changes were made: * Remove settings that just re-set default values, instead pointing to the website to retrieve defaults. * Simplify the errcheck rule to only include functions we actually need to ignore.
* update smoke alerts Signed-off-by: Robbie Lankford <robert.lankford@grafana.com> * filter pod variable with namespace Signed-off-by: Robbie Lankford <robert.lankford@grafana.com> * update cpu high alert duration to 1h Signed-off-by: Robbie Lankford <robert.lankford@grafana.com>
* upgrade to loki v2.4.2 Signed-off-by: Robbie Lankford <robert.lankford@grafana.com> * update CHANGELOG.md * rename cortex_ metrics to agent_dskit_ * update changelog
…#1293) * Fixes grafana-cold-storage#1281 * Update pkg/integrations/cadvisor/common.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * gofmt * Add tests for cadvisor config * Lint * simplify test, enhance comments, return nil on run * Default config slices to values expected by upstream cadvisor * Build flags to only run cadvisor tests when docker and network are present Co-authored-by: Robert Fratto <robert.fratto@grafana.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>
* Update install-agent-on-windows.md add quotes to URL silent install doc * Update install-agent-on-windows.md * Update install-agent-on-windows.md * Update install-agent-on-windows.md Fix linting error
* add bump-formula-pr workflow jobs Signed-off-by: Robbie Lankford <robert.lankford@grafana.com> * Update .github/workflows/bump-formula-pr.yml * Update .github/workflows/bump-formula-pr.yml * Update .github/workflows/bump-formula-pr.yml * Update .github/workflows/bump-formula-pr.yml * Update .github/workflows/bump-formula-pr.yml
* add consulagent_sd * add consulagent_sd
* Merge main -> dev.dynamic_configuration (#1270) * Update node_exporter dependency to v1.3.1 (#1228) * Add node_exporter to depcheck * update weaveworks/common dependency * map current release flags and changed defaults * documentation * revert accidental checkin * print out flags when node_exporter test fails to assist debugging * oops, i introduced some flags from master by mistake * Introduce experimental integrations revamp (#1198) * [dev.multiple-integrations] Enable present integrations by default, deprecate enabled field (#1062) * integrations: default to enabled by default * document deprecation of enabled * pkg/integrations: support *_configs field for integrations (#1130) Creates the basic code to unmarshal integrations from a YAML field called <integration name>_configs, which is a slice of that integration. Note that this is NOT wired up to the integrations manager yet, and trying to run the agent with more than one integration of the same type will likely cause problems. * [dev.multiple-integrations] Prototype new integrations subsystem (#1142) * wip: prototype new integrations subsystem * implement Controller with basic logic for Integration and UpdateIntegration * Implement HTTPIntegration for Controller * decouple controller and subsystem * don't have controller implement integration slightly less smelly now * multiplexer integration * rely on boilerplate for multiplexing for now generics would be nice here * remove multiplex_integration.go Also a little code smelly. Instead of having integrations that run other integrations, I'm going to fall back to having only one controller. * introduce Subsystem, unexport Controller start wiring up things to Subsystem * introduce v2 agent integration to use for testing * start wiring metrics integrations * rename Options to Globals call a spade a spade * add subsystem options to globals * remove dead code * metricsutils: calculate self-scraping based on globals * complete HTTP target API * working example with agent integration * appease the linter * don't return an error when context to cancel an integration is closed * once again i am asking the linter to forgive my typos * fix bug where labels from individual targets were getting dropped at the API endpoint * pkg/config: fix broken test * finish unit tests for integrations v2 controller * metricsutil/metricshandler_integration: make job name unique Before this change, the job name would have collided when using multiple instances of the same integration. * ensure that global subsystem labels are injected into targets * integrations/v2: Infer target hostname from SD API host (#1175) * [dev.multiple-integrations] integrations/v2: allow shimming between v1 and v2 integrations. (#1179) * integrations/v2: allow shimming between v1 and v2 integrations. Shimming is done by changing how the integration registration works; a new RegisterDynamic was added that allows for creating Configs at runtime. Here be dragons; this should be removed whenever we no longer have a need for it. * fix lint * pkg/integrations/v2: use "RegisterLegacy" instead of a generic mechanism * fine, I won't add the deprecation notice if it will make the linter sad * pkg/integrations: re-align (#1181) This commit reverts 69ba2ddfa9483cc8ac6e010dd7abccd319580c80 in favor of allowing the new subsystem to handle multiple instances of integrations. This commit also removes the wal_truncate_frequency field from integrations as it is the only field from old integrations that does not have a current counterpart. * [dev.multiple-integrations] Hide integrations/v2 behind a feature flag (#1185) * feature flag wip * dynamically switch between integrations v1 and v2 default to v1. * pkg/integrations/versionselector to file in pkg/config * pkg/config: fix defaults for Integrations * pkg/config: use more generic way to unmarshal differently based on flag * add missing godoc comment * more comments * switch to deferred unmarshaling * remove unused Config field * simplify completeUnmarshal * do not perform lazy deferred unmarshaling * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter * Merge main into dev.multiple-integrations (#1184) * Fix typo (#1141) * Traces: Improved pod association in PromSD processor (#1137) * Improve k8s pod association * Add tests * Changelog * typo * Add prom_sd_pod_association * Extend tests for pod associations * Docs for pod association config * Lint fixes * Move to unreleased * Add instrumentation recommendations * Remove uncessary constants * Improve tests * remote config with http(s) provider (#1143) * sample remote config code with http provider * use t.TempDir() in unit test * no need to clean up after T.TempDir() * use NewClientFromConfig and make caller responsible for calling SetDirectory * handle nil HTTPClientConfig * remove blank identifier assignment * pass basic auth command line flags for remote config * address pr nits * add expiremental flag * set loader inline * update changelog * add remote config section in docs * pr comment updates * announce patch releases for cve-2021-41090 (#1152) * Merge patch release to main (#1153) * Add secret type to sensitive values * Break out config tests to their own implementation. Also remove username has a sensitive value. * Update changelog * Fix failing test * Scrub secrets when marshaling instance configs * update for v0.21 * Updated changes from the merge. * Remove changelog * Scrub out receivers has ***receivers_scrubber***:null * obscure etcd/consul credentials * Update pkg/traces/config_test.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Update pkg/config/config.go * go fmt * Change to using custom object and return <secret> * Fix bad merge * [v0.21.2] toggle config endpoint (#19) * disable /-/config endpoint by default * disable scraping api get endpoint as well * fix new test * add test and rename flag Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Update version to v0.21.2 * Update defaults.go * fix /-/config endpoint * also fix non-pointer config bug * temporarily disable linting for release * fix lint errors Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter (#1162) * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter A recent change broke the usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME for the postgres_exporter. As the incorrect variable was checked in the if clause, it always raises an error. * changelog: keep feature -> enhancement -> bugfix order * postgres_exporter: add regression test Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix syntax error in Jsonnet logs helper method (#1174) Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * cAdvisor Integration (#1081) * Add cadvisor module * Begin creating common config for cadvisor * Don't export internal state * Finish config options for cadvisor * Set config options, and implement cAdvisor collectors * Linting * Buildflags for cadvisor only in linux * I R LEArN Build Tags * Don't zero value the zero value * Offload sketchy global var manipulation to the integrations Run func * Remove unused collectors * Lint * Create generic stub integration and use it for cadvisor * Lint * Final refactor of cAdvisor config for unsupported platforms. Pared down stub integrations. * Lint * Docs for cadvisor config * Update changelog * Update pkg/integrations/stub_integration.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Reorder changelog * Instance key clarity * Inclusive naming * Finish name changes Keep default disable metric list in sync with upstream Idiomatic golang * Hardcode disabled metrics for cadvisor Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Remove log-level flag from systemd unit file (#1177) * Upgrade to OTel v0.40.0 (#1176) * Upgrade to OTel v0.40.0 * Changelog * Add factories check * go mod tidy * config/features: create package to standardize experimental features (#1170) * config/features: create package to standardize experiemental features This commit introduces a new package, pkg/config/features, which allows defining a set of features and validating whether flags associated with those features are allowed to be set. Closes #1163 * update documentation (also s/enabled-features/enable-features) * Fix typo * Update pkg/config/features/features.go Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Revert "Merge main into dev.multiple-integrations (#1184)" (#1189) This reverts commit ad76ec54f9a781fc83d3e7001808c887f37833ff. * [dev.multiple-integrations] Revert breaking changes to existing integrations (#1191) * revert breaking changes to integrations v1 This commit reverts #1062 in favor of making breaking changes directly in integrations-next instead. The part of #1181 to remove `wal_truncate_frequency` has also been reverted. As part of this change, the enabled field is removed from the v2 common metrics configs, and v2 integrations can no longer be disabled. v2 integrations can only be disabled by removing them from the YAML. * integrations/v2: remove stale reference to ErrDisabled (fix typo too) * integrations/v2: bring in common config decoupling * [dev.multiple-integrations] Introduce autoscraper (#1195) * pkg/integrations/v2: introduce self-scraping * linting * [dev.multiple-integrations] Multiple instances of integrations (#1196) * multiple instances of integrations opt in relevant v1 integrations into supporting multiple instances * shims should check for instance key override * Document integrations-next (#1197) * document integrations-next * remove json tags since they make markdown unhappy * changelog * s/Run/RunIntegration * remove stale comment about integrations.controller purpose * create dedicated run method for instanceScraper * s/expoter/exporter/g * Document why an autoscrape.Scraper manages a set of per-instance scrapers * spell out prerequisite instead of pre-req * use go.uber.org/atomic to make the code a little easier to follow * remove started callback for running integration * use smaller interface for autoscrape Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Fix panic when using 'stdout' in automatic logging (#1233) * integrations-next: fix bug where v2 integrations were not being strictly unmarshaled (#1235) * Remove jsonnet vendor folders (#1222) * remove jsonnet vendor This adds all vendor folders into .gitignore and removes cached vendor files from the repository. Closes #1221 * Update scripts and instructions for jsonnet vendor removal * `make example-dashboards` will now also run `jb install` * k3d environment instructions now include `jb install` * smoke-test.bash will now run `jb install` prior to `tk apply` * Fix link to k3d example in DEVELOPERS.md (#1242) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix node_exporter upgrade docs (#1239) * Fix panic in automatic logging with stdout backend (#1243) * pkg/util: support custom yaml.Unmarshaler implementations for util.UnmarshalYAMLMerged (#1244) It's common for config types to have implement yaml.Unmarshaler for: * Applying defaults * Applying extra logic post-unmarshal If these config types were unmarshaled through util.UnmarshalYAMLMerged, the yaml.Unmarshaler implementation would never complete successfully, preventing the post-unmarshal logic from running. This issue was introduced in #1192, but went unnoticed until #1228 implemented yaml.Unmarshaler to perform field migrations. #1240 reported the issue. This commit fixes the bug by performing a second non-strict unmarshal to ensure that all input values unmarshal successfully, with the exception of unmarshal errors unrelated to unrecognized field names. This is hacky, but it's worthwhile noting that util.UnmarshalYAMLMerged is a temporary workaround needed for the integrations-next migration, and will eventually be removed. * Update k3d example grafana/grafonnet-lib version (#1246) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Create an e2e framework with support for running tests against k8s (#1234) * e2e: create an e2e framework with support for running tests against a k3d cluster * add new E2E drone job * E2E tests should pass when doing a release * sign drone.yml again * move e2e lint to different step that has golangci-lint installed * upgrade golangci-lint and go for e2e test * e2e: add gcc * E2E: install build-essential to get a working full gcc env * :( * e2e: support running from inside of docker * fix lint error * address review feedback * Operator: fix bug where /-/ready and /-/healthy always returned 404 (#1252) * operator: fix bug where /-/ready and /-/healthy always returned 404 controller-runtime must have at least one ready/healthy check for the endpoints to exist * fix lint error, use healthz.Ping * Make scraping-svc use the new `metrics:` key (#1259) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * update prometheus dependency (#1260) * corrected typo (#1265) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags (#1264) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Use :main tag for images in smoke test Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Set IMAGE_BRANCH_TAG env var in drone and actions pipelines Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove quotes from Makefile variable Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove force_release action Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * prepare for v0.22.0 release (#1266) * prepare for v0.22.0 release * remove E2E pipeline Co-authored-by: Robert Fratto <robert.fratto@grafana.com> Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Patrick Koenig <pkoenig10@gmail.com> Co-authored-by: DataPoints <langer.markus@gmail.com> * Main merge dynamic (#1305) * Update node_exporter dependency to v1.3.1 (#1228) * Add node_exporter to depcheck * update weaveworks/common dependency * map current release flags and changed defaults * documentation * revert accidental checkin * print out flags when node_exporter test fails to assist debugging * oops, i introduced some flags from master by mistake * Introduce experimental integrations revamp (#1198) * [dev.multiple-integrations] Enable present integrations by default, deprecate enabled field (#1062) * integrations: default to enabled by default * document deprecation of enabled * pkg/integrations: support *_configs field for integrations (#1130) Creates the basic code to unmarshal integrations from a YAML field called <integration name>_configs, which is a slice of that integration. Note that this is NOT wired up to the integrations manager yet, and trying to run the agent with more than one integration of the same type will likely cause problems. * [dev.multiple-integrations] Prototype new integrations subsystem (#1142) * wip: prototype new integrations subsystem * implement Controller with basic logic for Integration and UpdateIntegration * Implement HTTPIntegration for Controller * decouple controller and subsystem * don't have controller implement integration slightly less smelly now * multiplexer integration * rely on boilerplate for multiplexing for now generics would be nice here * remove multiplex_integration.go Also a little code smelly. Instead of having integrations that run other integrations, I'm going to fall back to having only one controller. * introduce Subsystem, unexport Controller start wiring up things to Subsystem * introduce v2 agent integration to use for testing * start wiring metrics integrations * rename Options to Globals call a spade a spade * add subsystem options to globals * remove dead code * metricsutils: calculate self-scraping based on globals * complete HTTP target API * working example with agent integration * appease the linter * don't return an error when context to cancel an integration is closed * once again i am asking the linter to forgive my typos * fix bug where labels from individual targets were getting dropped at the API endpoint * pkg/config: fix broken test * finish unit tests for integrations v2 controller * metricsutil/metricshandler_integration: make job name unique Before this change, the job name would have collided when using multiple instances of the same integration. * ensure that global subsystem labels are injected into targets * integrations/v2: Infer target hostname from SD API host (#1175) * [dev.multiple-integrations] integrations/v2: allow shimming between v1 and v2 integrations. (#1179) * integrations/v2: allow shimming between v1 and v2 integrations. Shimming is done by changing how the integration registration works; a new RegisterDynamic was added that allows for creating Configs at runtime. Here be dragons; this should be removed whenever we no longer have a need for it. * fix lint * pkg/integrations/v2: use "RegisterLegacy" instead of a generic mechanism * fine, I won't add the deprecation notice if it will make the linter sad * pkg/integrations: re-align (#1181) This commit reverts 69ba2ddfa9483cc8ac6e010dd7abccd319580c80 in favor of allowing the new subsystem to handle multiple instances of integrations. This commit also removes the wal_truncate_frequency field from integrations as it is the only field from old integrations that does not have a current counterpart. * [dev.multiple-integrations] Hide integrations/v2 behind a feature flag (#1185) * feature flag wip * dynamically switch between integrations v1 and v2 default to v1. * pkg/integrations/versionselector to file in pkg/config * pkg/config: fix defaults for Integrations * pkg/config: use more generic way to unmarshal differently based on flag * add missing godoc comment * more comments * switch to deferred unmarshaling * remove unused Config field * simplify completeUnmarshal * do not perform lazy deferred unmarshaling * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter * Merge main into dev.multiple-integrations (#1184) * Fix typo (#1141) * Traces: Improved pod association in PromSD processor (#1137) * Improve k8s pod association * Add tests * Changelog * typo * Add prom_sd_pod_association * Extend tests for pod associations * Docs for pod association config * Lint fixes * Move to unreleased * Add instrumentation recommendations * Remove uncessary constants * Improve tests * remote config with http(s) provider (#1143) * sample remote config code with http provider * use t.TempDir() in unit test * no need to clean up after T.TempDir() * use NewClientFromConfig and make caller responsible for calling SetDirectory * handle nil HTTPClientConfig * remove blank identifier assignment * pass basic auth command line flags for remote config * address pr nits * add expiremental flag * set loader inline * update changelog * add remote config section in docs * pr comment updates * announce patch releases for cve-2021-41090 (#1152) * Merge patch release to main (#1153) * Add secret type to sensitive values * Break out config tests to their own implementation. Also remove username has a sensitive value. * Update changelog * Fix failing test * Scrub secrets when marshaling instance configs * update for v0.21 * Updated changes from the merge. * Remove changelog * Scrub out receivers has ***receivers_scrubber***:null * obscure etcd/consul credentials * Update pkg/traces/config_test.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Update pkg/config/config.go * go fmt * Change to using custom object and return <secret> * Fix bad merge * [v0.21.2] toggle config endpoint (#19) * disable /-/config endpoint by default * disable scraping api get endpoint as well * fix new test * add test and rename flag Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Update version to v0.21.2 * Update defaults.go * fix /-/config endpoint * also fix non-pointer config bug * temporarily disable linting for release * fix lint errors Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter (#1162) * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter A recent change broke the usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME for the postgres_exporter. As the incorrect variable was checked in the if clause, it always raises an error. * changelog: keep feature -> enhancement -> bugfix order * postgres_exporter: add regression test Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix syntax error in Jsonnet logs helper method (#1174) Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * cAdvisor Integration (#1081) * Add cadvisor module * Begin creating common config for cadvisor * Don't export internal state * Finish config options for cadvisor * Set config options, and implement cAdvisor collectors * Linting * Buildflags for cadvisor only in linux * I R LEArN Build Tags * Don't zero value the zero value * Offload sketchy global var manipulation to the integrations Run func * Remove unused collectors * Lint * Create generic stub integration and use it for cadvisor * Lint * Final refactor of cAdvisor config for unsupported platforms. Pared down stub integrations. * Lint * Docs for cadvisor config * Update changelog * Update pkg/integrations/stub_integration.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Reorder changelog * Instance key clarity * Inclusive naming * Finish name changes Keep default disable metric list in sync with upstream Idiomatic golang * Hardcode disabled metrics for cadvisor Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Remove log-level flag from systemd unit file (#1177) * Upgrade to OTel v0.40.0 (#1176) * Upgrade to OTel v0.40.0 * Changelog * Add factories check * go mod tidy * config/features: create package to standardize experimental features (#1170) * config/features: create package to standardize experiemental features This commit introduces a new package, pkg/config/features, which allows defining a set of features and validating whether flags associated with those features are allowed to be set. Closes #1163 * update documentation (also s/enabled-features/enable-features) * Fix typo * Update pkg/config/features/features.go Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Revert "Merge main into dev.multiple-integrations (#1184)" (#1189) This reverts commit ad76ec54f9a781fc83d3e7001808c887f37833ff. * [dev.multiple-integrations] Revert breaking changes to existing integrations (#1191) * revert breaking changes to integrations v1 This commit reverts #1062 in favor of making breaking changes directly in integrations-next instead. The part of #1181 to remove `wal_truncate_frequency` has also been reverted. As part of this change, the enabled field is removed from the v2 common metrics configs, and v2 integrations can no longer be disabled. v2 integrations can only be disabled by removing them from the YAML. * integrations/v2: remove stale reference to ErrDisabled (fix typo too) * integrations/v2: bring in common config decoupling * [dev.multiple-integrations] Introduce autoscraper (#1195) * pkg/integrations/v2: introduce self-scraping * linting * [dev.multiple-integrations] Multiple instances of integrations (#1196) * multiple instances of integrations opt in relevant v1 integrations into supporting multiple instances * shims should check for instance key override * Document integrations-next (#1197) * document integrations-next * remove json tags since they make markdown unhappy * changelog * s/Run/RunIntegration * remove stale comment about integrations.controller purpose * create dedicated run method for instanceScraper * s/expoter/exporter/g * Document why an autoscrape.Scraper manages a set of per-instance scrapers * spell out prerequisite instead of pre-req * use go.uber.org/atomic to make the code a little easier to follow * remove started callback for running integration * use smaller interface for autoscrape Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Fix panic when using 'stdout' in automatic logging (#1233) * integrations-next: fix bug where v2 integrations were not being strictly unmarshaled (#1235) * Remove jsonnet vendor folders (#1222) * remove jsonnet vendor This adds all vendor folders into .gitignore and removes cached vendor files from the repository. Closes #1221 * Update scripts and instructions for jsonnet vendor removal * `make example-dashboards` will now also run `jb install` * k3d environment instructions now include `jb install` * smoke-test.bash will now run `jb install` prior to `tk apply` * Fix link to k3d example in DEVELOPERS.md (#1242) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix node_exporter upgrade docs (#1239) * Fix panic in automatic logging with stdout backend (#1243) * pkg/util: support custom yaml.Unmarshaler implementations for util.UnmarshalYAMLMerged (#1244) It's common for config types to have implement yaml.Unmarshaler for: * Applying defaults * Applying extra logic post-unmarshal If these config types were unmarshaled through util.UnmarshalYAMLMerged, the yaml.Unmarshaler implementation would never complete successfully, preventing the post-unmarshal logic from running. This issue was introduced in #1192, but went unnoticed until #1228 implemented yaml.Unmarshaler to perform field migrations. #1240 reported the issue. This commit fixes the bug by performing a second non-strict unmarshal to ensure that all input values unmarshal successfully, with the exception of unmarshal errors unrelated to unrecognized field names. This is hacky, but it's worthwhile noting that util.UnmarshalYAMLMerged is a temporary workaround needed for the integrations-next migration, and will eventually be removed. * Update k3d example grafana/grafonnet-lib version (#1246) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Create an e2e framework with support for running tests against k8s (#1234) * e2e: create an e2e framework with support for running tests against a k3d cluster * add new E2E drone job * E2E tests should pass when doing a release * sign drone.yml again * move e2e lint to different step that has golangci-lint installed * upgrade golangci-lint and go for e2e test * e2e: add gcc * E2E: install build-essential to get a working full gcc env * :( * e2e: support running from inside of docker * fix lint error * address review feedback * Operator: fix bug where /-/ready and /-/healthy always returned 404 (#1252) * operator: fix bug where /-/ready and /-/healthy always returned 404 controller-runtime must have at least one ready/healthy check for the endpoints to exist * fix lint error, use healthz.Ping * Make scraping-svc use the new `metrics:` key (#1259) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * update prometheus dependency (#1260) * corrected typo (#1265) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags (#1264) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Use :main tag for images in smoke test Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Set IMAGE_BRANCH_TAG env var in drone and actions pipelines Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove quotes from Makefile variable Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove force_release action Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * prepare for v0.22.0 release (#1266) * prepare for v0.22.0 release * remove E2E pipeline * Add basic testing framework for operator (#1268) * remove dedicated go.mod for e2e/ * move e2e/k8s to pkg/util/k8s * Migrate operator tests to pkg/util/k8s * remove dedicated e2e tests * allow skipping TestCluster in pkg/util/k8s * remove e2e/ * fix bad merge * fix order of make env args for windows * actually declare referenced docker volume * introduce pkg/util/subset for asserting subset of objects * refactor operator so it's testable * define basic integration test for operator * fix lint errors * fix invalid address in operator test config * Update release-note.md (#1267) * Set scrape User-Agent header during init (#1274) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Upgrade to Go 1.17 (#1278) * Upgrade to 1.17.6 in go.mod and Dockerfiles * Update CHANGELOG.md to mention the update * Update Go version in drone/actions pipelines * Update go.mod, go.sum files via * Re-sign drone.yml * Remove leading newline causing drone build to fail * Bump golangci-lint image to a version using Go 1.17 * Re-attempt to solve linter issue with new golangci-lint image * Remove suffix of exclude rules * Clean previous Go version before unpacking Go 1.17 * Also clean up previous Go versions in other steps * fix typo (#1284) * Use custom Go version in agent-operator Dockerfile (#1286) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * pkg/operator: refactor resource hierarchy discovery (#1271) * pkg/operator: refactor resource hierarchy discovery This commit moves common logic related to discovering the resource hierarchy to pkg/operator/hierarchy. This new package requires less boilerplate, which the reconciler is updated to take advantage of. * remove unused code * test construction of resource hierarchy * add missing build constraints * small extra cleanup to use pointer package * review feedback * update agent-build-image for go 1.17 (#1287) (also use a consistent base image tag instead of latest) * Skip non-ready entries when listing instances (#1289) * Skip non-ready instances in LoadInstances() Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add changelog entry Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix panic in prom_sd_processor when address is empty (#1279) * Fix panic in prom_sd_processor when address is empty * Fix panic in prom_sd_processor when address is empty * Fix docs * Add test case * Lint * Move to unreleased * Operator: generate proxy_url for remote_write (#1298) * operator: generate proxy_url for remote_write * fix weird indentation in test Co-authored-by: Robert Fratto <robert.fratto@grafana.com> Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Patrick Koenig <pkoenig10@gmail.com> Co-authored-by: DataPoints <langer.markus@gmail.com> Co-authored-by: Alex <52292902+alexrudd2@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Main merge dynamic (#1307) * Update node_exporter dependency to v1.3.1 (#1228) * Add node_exporter to depcheck * update weaveworks/common dependency * map current release flags and changed defaults * documentation * revert accidental checkin * print out flags when node_exporter test fails to assist debugging * oops, i introduced some flags from master by mistake * Introduce experimental integrations revamp (#1198) * [dev.multiple-integrations] Enable present integrations by default, deprecate enabled field (#1062) * integrations: default to enabled by default * document deprecation of enabled * pkg/integrations: support *_configs field for integrations (#1130) Creates the basic code to unmarshal integrations from a YAML field called <integration name>_configs, which is a slice of that integration. Note that this is NOT wired up to the integrations manager yet, and trying to run the agent with more than one integration of the same type will likely cause problems. * [dev.multiple-integrations] Prototype new integrations subsystem (#1142) * wip: prototype new integrations subsystem * implement Controller with basic logic for Integration and UpdateIntegration * Implement HTTPIntegration for Controller * decouple controller and subsystem * don't have controller implement integration slightly less smelly now * multiplexer integration * rely on boilerplate for multiplexing for now generics would be nice here * remove multiplex_integration.go Also a little code smelly. Instead of having integrations that run other integrations, I'm going to fall back to having only one controller. * introduce Subsystem, unexport Controller start wiring up things to Subsystem * introduce v2 agent integration to use for testing * start wiring metrics integrations * rename Options to Globals call a spade a spade * add subsystem options to globals * remove dead code * metricsutils: calculate self-scraping based on globals * complete HTTP target API * working example with agent integration * appease the linter * don't return an error when context to cancel an integration is closed * once again i am asking the linter to forgive my typos * fix bug where labels from individual targets were getting dropped at the API endpoint * pkg/config: fix broken test * finish unit tests for integrations v2 controller * metricsutil/metricshandler_integration: make job name unique Before this change, the job name would have collided when using multiple instances of the same integration. * ensure that global subsystem labels are injected into targets * integrations/v2: Infer target hostname from SD API host (#1175) * [dev.multiple-integrations] integrations/v2: allow shimming between v1 and v2 integrations. (#1179) * integrations/v2: allow shimming between v1 and v2 integrations. Shimming is done by changing how the integration registration works; a new RegisterDynamic was added that allows for creating Configs at runtime. Here be dragons; this should be removed whenever we no longer have a need for it. * fix lint * pkg/integrations/v2: use "RegisterLegacy" instead of a generic mechanism * fine, I won't add the deprecation notice if it will make the linter sad * pkg/integrations: re-align (#1181) This commit reverts 69ba2ddfa9483cc8ac6e010dd7abccd319580c80 in favor of allowing the new subsystem to handle multiple instances of integrations. This commit also removes the wal_truncate_frequency field from integrations as it is the only field from old integrations that does not have a current counterpart. * [dev.multiple-integrations] Hide integrations/v2 behind a feature flag (#1185) * feature flag wip * dynamically switch between integrations v1 and v2 default to v1. * pkg/integrations/versionselector to file in pkg/config * pkg/config: fix defaults for Integrations * pkg/config: use more generic way to unmarshal differently based on flag * add missing godoc comment * more comments * switch to deferred unmarshaling * remove unused Config field * simplify completeUnmarshal * do not perform lazy deferred unmarshaling * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter * Merge main into dev.multiple-integrations (#1184) * Fix typo (#1141) * Traces: Improved pod association in PromSD processor (#1137) * Improve k8s pod association * Add tests * Changelog * typo * Add prom_sd_pod_association * Extend tests for pod associations * Docs for pod association config * Lint fixes * Move to unreleased * Add instrumentation recommendations * Remove uncessary constants * Improve tests * remote config with http(s) provider (#1143) * sample remote config code with http provider * use t.TempDir() in unit test * no need to clean up after T.TempDir() * use NewClientFromConfig and make caller responsible for calling SetDirectory * handle nil HTTPClientConfig * remove blank identifier assignment * pass basic auth command line flags for remote config * address pr nits * add expiremental flag * set loader inline * update changelog * add remote config section in docs * pr comment updates * announce patch releases for cve-2021-41090 (#1152) * Merge patch release to main (#1153) * Add secret type to sensitive values * Break out config tests to their own implementation. Also remove username has a sensitive value. * Update changelog * Fix failing test * Scrub secrets when marshaling instance configs * update for v0.21 * Updated changes from the merge. * Remove changelog * Scrub out receivers has ***receivers_scrubber***:null * obscure etcd/consul credentials * Update pkg/traces/config_test.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Update pkg/config/config.go * go fmt * Change to using custom object and return <secret> * Fix bad merge * [v0.21.2] toggle config endpoint (#19) * disable /-/config endpoint by default * disable scraping api get endpoint as well * fix new test * add test and rename flag Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Update version to v0.21.2 * Update defaults.go * fix /-/config endpoint * also fix non-pointer config bug * temporarily disable linting for release * fix lint errors Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter (#1162) * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter A recent change broke the usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME for the postgres_exporter. As the incorrect variable was checked in the if clause, it always raises an error. * changelog: keep feature -> enhancement -> bugfix order * postgres_exporter: add regression test Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix syntax error in Jsonnet logs helper method (#1174) Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * cAdvisor Integration (#1081) * Add cadvisor module * Begin creating common config for cadvisor * Don't export internal state * Finish config options for cadvisor * Set config options, and implement cAdvisor collectors * Linting * Buildflags for cadvisor only in linux * I R LEArN Build Tags * Don't zero value the zero value * Offload sketchy global var manipulation to the integrations Run func * Remove unused collectors * Lint * Create generic stub integration and use it for cadvisor * Lint * Final refactor of cAdvisor config for unsupported platforms. Pared down stub integrations. * Lint * Docs for cadvisor config * Update changelog * Update pkg/integrations/stub_integration.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Reorder changelog * Instance key clarity * Inclusive naming * Finish name changes Keep default disable metric list in sync with upstream Idiomatic golang * Hardcode disabled metrics for cadvisor Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Remove log-level flag from systemd unit file (#1177) * Upgrade to OTel v0.40.0 (#1176) * Upgrade to OTel v0.40.0 * Changelog * Add factories check * go mod tidy * config/features: create package to standardize experimental features (#1170) * config/features: create package to standardize experiemental features This commit introduces a new package, pkg/config/features, which allows defining a set of features and validating whether flags associated with those features are allowed to be set. Closes #1163 * update documentation (also s/enabled-features/enable-features) * Fix typo * Update pkg/config/features/features.go Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Revert "Merge main into dev.multiple-integrations (#1184)" (#1189) This reverts commit ad76ec54f9a781fc83d3e7001808c887f37833ff. * [dev.multiple-integrations] Revert breaking changes to existing integrations (#1191) * revert breaking changes to integrations v1 This commit reverts #1062 in favor of making breaking changes directly in integrations-next instead. The part of #1181 to remove `wal_truncate_frequency` has also been reverted. As part of this change, the enabled field is removed from the v2 common metrics configs, and v2 integrations can no longer be disabled. v2 integrations can only be disabled by removing them from the YAML. * integrations/v2: remove stale reference to ErrDisabled (fix typo too) * integrations/v2: bring in common config decoupling * [dev.multiple-integrations] Introduce autoscraper (#1195) * pkg/integrations/v2: introduce self-scraping * linting * [dev.multiple-integrations] Multiple instances of integrations (#1196) * multiple instances of integrations opt in relevant v1 integrations into supporting multiple instances * shims should check for instance key override * Document integrations-next (#1197) * document integrations-next * remove json tags since they make markdown unhappy * changelog * s/Run/RunIntegration * remove stale comment about integrations.controller purpose * create dedicated run method for instanceScraper * s/expoter/exporter/g * Document why an autoscrape.Scraper manages a set of per-instance scrapers * spell out prerequisite instead of pre-req * use go.uber.org/atomic to make the code a little easier to follow * remove started callback for running integration * use smaller interface for autoscrape Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * Fix panic when using 'stdout' in automatic logging (#1233) * integrations-next: fix bug where v2 integrations were not being strictly unmarshaled (#1235) * Remove jsonnet vendor folders (#1222) * remove jsonnet vendor This adds all vendor folders into .gitignore and removes cached vendor files from the repository. Closes #1221 * Update scripts and instructions for jsonnet vendor removal * `make example-dashboards` will now also run `jb install` * k3d environment instructions now include `jb install` * smoke-test.bash will now run `jb install` prior to `tk apply` * Fix link to k3d example in DEVELOPERS.md (#1242) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix node_exporter upgrade docs (#1239) * Fix panic in automatic logging with stdout backend (#1243) * pkg/util: support custom yaml.Unmarshaler implementations for util.UnmarshalYAMLMerged (#1244) It's common for config types to have implement yaml.Unmarshaler for: * Applying defaults * Applying extra logic post-unmarshal If these config types were unmarshaled through util.UnmarshalYAMLMerged, the yaml.Unmarshaler implementation would never complete successfully, preventing the post-unmarshal logic from running. This issue was introduced in #1192, but went unnoticed until #1228 implemented yaml.Unmarshaler to perform field migrations. #1240 reported the issue. This commit fixes the bug by performing a second non-strict unmarshal to ensure that all input values unmarshal successfully, with the exception of unmarshal errors unrelated to unrecognized field names. This is hacky, but it's worthwhile noting that util.UnmarshalYAMLMerged is a temporary workaround needed for the integrations-next migration, and will eventually be removed. * Update k3d example grafana/grafonnet-lib version (#1246) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Create an e2e framework with support for running tests against k8s (#1234) * e2e: create an e2e framework with support for running tests against a k3d cluster * add new E2E drone job * E2E tests should pass when doing a release * sign drone.yml again * move e2e lint to different step that has golangci-lint installed * upgrade golangci-lint and go for e2e test * e2e: add gcc * E2E: install build-essential to get a working full gcc env * :( * e2e: support running from inside of docker * fix lint error * address review feedback * Operator: fix bug where /-/ready and /-/healthy always returned 404 (#1252) * operator: fix bug where /-/ready and /-/healthy always returned 404 controller-runtime must have at least one ready/healthy check for the endpoints to exist * fix lint error, use healthz.Ping * Make scraping-svc use the new `metrics:` key (#1259) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * update prometheus dependency (#1260) * corrected typo (#1265) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags (#1264) * Use RELEASE_TAG to choose between `:main` and `:latest` docker tags Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Use :main tag for images in smoke test Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Set IMAGE_BRANCH_TAG env var in drone and actions pipelines Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove quotes from Makefile variable Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove force_release action Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * prepare for v0.22.0 release (#1266) * prepare for v0.22.0 release * remove E2E pipeline * Add basic testing framework for operator (#1268) * remove dedicated go.mod for e2e/ * move e2e/k8s to pkg/util/k8s * Migrate operator tests to pkg/util/k8s * remove dedicated e2e tests * allow skipping TestCluster in pkg/util/k8s * remove e2e/ * fix bad merge * fix order of make env args for windows * actually declare referenced docker volume * introduce pkg/util/subset for asserting subset of objects * refactor operator so it's testable * define basic integration test for operator * fix lint errors * fix invalid address in operator test config * Update release-note.md (#1267) * Set scrape User-Agent header during init (#1274) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Upgrade to Go 1.17 (#1278) * Upgrade to 1.17.6 in go.mod and Dockerfiles * Update CHANGELOG.md to mention the update * Update Go version in drone/actions pipelines * Update go.mod, go.sum files via * Re-sign drone.yml * Remove leading newline causing drone build to fail * Bump golangci-lint image to a version using Go 1.17 * Re-attempt to solve linter issue with new golangci-lint image * Remove suffix of exclude rules * Clean previous Go version before unpacking Go 1.17 * Also clean up previous Go versions in other steps * fix typo (#1284) * Use custom Go version in agent-operator Dockerfile (#1286) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * pkg/operator: refactor resource hierarchy discovery (#1271) * pkg/operator: refactor resource hierarchy discovery This commit moves common logic related to discovering the resource hierarchy to pkg/operator/hierarchy. This new package requires less boilerplate, which the reconciler is updated to take advantage of. * remove unused code * test construction of resource hierarchy * add missing build constraints * small extra cleanup to use pointer package * review feedback * update agent-build-image for go 1.17 (#1287) (also use a consistent base image tag instead of latest) * Skip non-ready entries when listing instances (#1289) * Skip non-ready instances in LoadInstances() Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add changelog entry Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix panic in prom_sd_processor when address is empty (#1279) * Fix panic in prom_sd_processor when address is empty * Fix panic in prom_sd_processor when address is empty * Fix docs * Add test case * Lint * Move to unreleased * Operator: generate proxy_url for remote_write (#1298) * operator: generate proxy_url for remote_write * fix weird indentation in test Co-authored-by: Robert Fratto <robert.fratto@grafana.com> Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Patrick Koenig <pkoenig10@gmail.com> Co-authored-by: DataPoints <langer.markus@gmail.com> Co-authored-by: Alex <52292902+alexrudd2@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * [RFC] Dynamic Documentation (#1308) * Documentation and feature flag support. Part 1 of many. * Fix linting * Documentation * More documentation * MOAR documentation * Update overall readme * fix typo * Update docs/configuration/dynamic-config.md Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * fix typos and add additional comments * Feedback from PR * Allow overrides of documentation Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Code to support dynamic configuration. (#1360) * Code to support dynamic configuration. * Fix linting errors * Fix issue with examples and add template parse * Fix windows issues * Simplify tests * Respond to PR feedback * Simplify tests and remove setting default * Move to unexported configshim * Export configshims members for testing * Simplify singleton checking * Add error around expand var * sever -> server * Better error code * Fix error when unmarshalling yaml which allowed unexporting fields * Switch to using require.noerror instead of assert. * Split tests into two sections and remove redundant tests * Verbiage cleanup and var renaming * PR feedback * PR feedback * Move singleton check to the controller instead * Cleanup the read file code to MUCH more compact code. * Remove EOF * PR feedback from session with rfratto, lots of changes to simplify the code * Fix misc feedback from PR * Dyn main merge (#1423) * Update node_exporter dependency to v1.3.1 (#1228) * Add node_exporter to depcheck * update weaveworks/common dependency * map current release flags and changed defaults * documentation * revert accidental checkin * print out flags when node_exporter test fails to assist debugging * oops, i introduced some flags from master by mistake * Introduce experimental integrations revamp (#1198) * [dev.multiple-integrations] Enable present integrations by default, deprecate enabled field (#1062) * integrations: default to enabled by default * document deprecation of enabled * pkg/integrations: support *_configs field for integrations (#1130) Creates the basic code to unmarshal integrations from a YAML field called <integration name>_configs, which is a slice of that integration. Note that this is NOT wired up to the integrations manager yet, and trying to run the agent with more than one integration of the same type will likely cause problems. * [dev.multiple-integrations] Prototype new integrations subsystem (#1142) * wip: prototype new integrations subsystem * implement Controller with basic logic for Integration and UpdateIntegration * Implement HTTPIntegration for Controller * decouple controller and subsystem * don't have controller implement integration slightly less smelly now * multiplexer integration * rely on boilerplate for multiplexing for now generics would be nice here * remove multiplex_integration.go Also a little code smelly. Instead of having integrations that run other integrations, I'm going to fall back to having only one controller. * introduce Subsystem, unexport Controller start wiring up things to Subsystem * introduce v2 agent integration to use for testing * start wiring metrics integrations * rename Options to Globals call a spade a spade * add subsystem options to globals * remove dead code * metricsutils: calculate self-scraping based on globals * complete HTTP target API * working example with agent integration * appease the linter * don't return an error when context to cancel an integration is closed * once again i am asking the linter to forgive my typos * fix bug where labels from individual targets were getting dropped at the API endpoint * pkg/config: fix broken test * finish unit tests for integrations v2 controller * metricsutil/metricshandler_integration: make job name unique Before this change, the job name would have collided when using multiple instances of the same integration. * ensure that global subsystem labels are injected into targets * integrations/v2: Infer target hostname from SD API host (#1175) * [dev.multiple-integrations] integrations/v2: allow shimming between v1 and v2 integrations. (#1179) * integrations/v2: allow shimming between v1 and v2 integrations. Shimming is done by changing how the integration registration works; a new RegisterDynamic was added that allows for creating Configs at runtime. Here be dragons; this should be removed whenever we no longer have a need for it. * fix lint * pkg/integrations/v2: use "RegisterLegacy" instead of a generic mechanism * fine, I won't add the deprecation notice if it will make the linter sad * pkg/integrations: re-align (#1181) This commit reverts 69ba2ddfa9483cc8ac6e010dd7abccd319580c80 in favor of allowing the new subsystem to handle multiple instances of integrations. This commit also removes the wal_truncate_frequency field from integrations as it is the only field from old integrations that does not have a current counterpart. * [dev.multiple-integrations] Hide integrations/v2 behind a feature flag (#1185) * feature flag wip * dynamically switch between integrations v1 and v2 default to v1. * pkg/integrations/versionselector to file in pkg/config * pkg/config: fix defaults for Integrations * pkg/config: use more generic way to unmarshal differently based on flag * add missing godoc comment * more comments * switch to deferred unmarshaling * remove unused Config field * simplify completeUnmarshal * do not perform lazy deferred unmarshaling * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter * Merge main into dev.multiple-integrations (#1184) * Fix typo (#1141) * Traces: Improved pod association in PromSD processor (#1137) * Improve k8s pod association * Add tests * Changelog * typo * Add prom_sd_pod_association * Extend tests for pod associations * Docs for pod association config * Lint fixes * Move to unreleased * Add instrumentation recommendations * Remove uncessary constants * Improve tests * remote config with http(s) provider (#1143) * sample remote config code with http provider * use t.TempDir() in unit test * no need to clean up after T.TempDir() * use NewClientFromConfig and make caller responsible for calling SetDirectory * handle nil HTTPClientConfig * remove blank identifier assignment * pass basic auth command line flags for remote config * address pr nits * add expiremental flag * set loader inline * update changelog * add remote config section in docs * pr comment updates * announce patch releases for cve-2021-41090 (#1152) * Merge patch release to main (#1153) * Add secret type to sensitive values * Break out config tests to their own implementation. Also remove username has a sensitive value. * Update changelog * Fix failing test * Scrub secrets when marshaling instance configs * update for v0.21 * Updated changes from the merge. * Remove changelog * Scrub out receivers has ***receivers_scrubber***:null * obscure etcd/consul credentials * Update pkg/traces/config_test.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Update pkg/config/config.go * go fmt * Change to using custom object and return <secret> * Fix bad merge * [v0.21.2] toggle config endpoint (#19) * disable /-/config endpoint by default * disable scraping api get endpoint as well * fix new test * add test and rename flag Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Update version to v0.21.2 * Update defaults.go * fix /-/config endpoint * also fix non-pointer config bug * temporarily disable linting for release * fix lint errors Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter (#1162) * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter A recent change broke the usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME for the postgres_exporter. As the incorrect variable was checked in the if clause, it always raises an error. * changelog: keep feature -> enhancement -> bugfix order * postgres_exporter: add regression test Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix syntax error in Jsonnet logs helper method (#1174) Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * cAdvisor Integration (#1081) * Add cadvisor module * Begin creating common config for cadvisor * Don't export internal state * Finish config options for cadvisor * Set config options, and implement cAdvisor collectors * Linting * Buildflags for cadvisor only in linux * I R LEArN Build Tags * Don't zero value the zero value * Offload sketchy global var manipulation to the integrations Run func * Remove unused collectors * Lint * Create generic stub integration and use it for cadvisor * Lint * Final refactor of cAdvisor config for unsupported platforms. Pared down stub in…
…1446) * document stability status and mark specific things as stable/beta/experiemntal * document stability transition * consistent casing * instance sharing is probably stable, it's been around long enough
* Parse const_labels correctly in exporter * range over key and value Co-authored-by: Mario <mariorvinas@gmail.com> * no need to use a pointer * add test with a spanmetrics config * add to changelog * fix lint * revert autoformater changes Co-authored-by: Mario <mariorvinas@gmail.com>
* add developer/contributor documentation * link to developer/ from docs/README.md * contributing.md: discuss PR size * ignore release candidates when bumping homebrew formula * Only update the :latest image tag when tagging a non-release candidate
…e#1452) * increase swap space on windows for running tests * Try windows-latest windows-latest had memory issues more consistently than windows-2019; if this fix works for windows-latest too it's a good sign. * Fix ignored tags in bump-formula-pr It's invalid to list both tags and ignore-tags, and you have to use the ! syntax to ignore a set of tags instead.
…rage#1449) * remove duplicate suffix from calls_total spanmetric * get data points based on data type
This fixes a race condition where the Prometheus Discovery Manager is updated at the same time as processing a new set of scrape targets.
* add container restarts alert * drop for so alert fires immediately
…#1456) The various docker-compose.*.yaml files have been replaced with a single docker-compose.yaml file. By default, this file will spawn and expose ports for Loki, Grafana, Tempo, and Cortex so telemetry data can be stored and visualized. This allows developers to bring up dependencies needed for testing a local Grafana Agent process, useful for validating code changes. Version 3 of Docker Compose is used for profile support, allowing docker-compose.yaml to optionally run extra services: * The `agent` profile will also run a Grafana Agent that collects metrics from itself and sends them to Cortex. * The `integrations` profile will run example services that integrations can target, such as MySQL and Consul. Each of these `integrations` services can also be enabled independently by using the profile name matching the relevant integration name (i.e., the `mysqld_exporter` profile enables the MySQL service). All integration services will expose their ports for a local agent to connect to.
… <name>_configs (grafana-cold-storage#1463) This relaxes the restrictions on how integrations are unmarshaled so that any integration, regardless of type, can be unmarshaled as an JSON object or an array. This makes configurations like this technically valid: integrations: node_exporter_configs: - {} Such a configuration would not be recommended to users, but as there is only one defined instance of node_exporter, it still satisfies the singleton requirements. It is unlikely that we would document this is possible. Introducing this change makes it *much* easier for the operator to generate configs for integrations, as we no longer have to switch between the _configs and non-_configs key based on the integration being generated.
…age#1473) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
The integrations-next subsystem invokes controller.Handler immediately after controller.UpdateController, which means that the controller's list of integrations must be maintained synchronously rather than in the background. If this doesn't happen, the generated handler may be missing some integrations. Fixes grafana-cold-storage#1455
This restores the ordering we try to keep for our CHANGELOG, and adds a note into the pull_request_template for contributors to retain that order. I also did some formatting and reclassified/rephrased some entries.
…rage#1476) * pkg/server: replace pkg/util/server with pkg/server pkg/server is a new top-level subsystem which exposes an HTTP and gRPC server. It was forked off of weaveworks/common/server with direct support for reloading settings we wish to keep as dynamic. The following YAML fields are intended to be dynamically updatable: * server.log_level * server.log_format * server.http_tls_config * server.grpc_tls_config Other fields are intended to be static and should only be set via command line flag. Their YAML counterparts are now DEPRECATED and will be removed in v0.26.0. As of this commit, reloading the config file will fail if a non-dynamic field has changed in the server block. Closes grafana-cold-storage#1336. Flag changes ------------ Many flag names corresponding to the server have changed. The following flags are NEW: * `-server.http.enable-tls` * `-server.grpc.enable-tls` * `-server.http.address` * `-server.grpc.address` The `-server.*-enable-tls` flags must now be passed to enable TLS for HTTP and gRPC respectively. This is a change from the previous behavior of dynamically enabling TLS when a certificate pair is provided. `-server.http.address` and `-server.grpc.address` are intended to be `host:port` addresses on which to listen for HTTP and gRPC traffic. They replace the `-server.*-listen-address` and `-server.*-listen-port` flags. The following flags have been RENAMED: * `-server.log.source-ips.enabled` (renamed from `-server.log-source-ips-enabled`) * `-server.log.source-ips.header` (renamed from `-server.log-source-ips-header`) * `-server.log.source-ips.regex` (renamed from `-server.log-source-ips-regex`) * `-server.http.network` (renamed from `-server.http-listen-network`) * `-server.http.conn-limit` (renamed from `-server.http-conn-limit`) * `-server.http.read-timeout` (renamed from `-server.http-read-timeout`) * `-server.http.write-timeout` (renamed from `-server.http-write-timeout`) * `-server.http.idle-timeout` (renamed from `-server.http-idle-timeout`) * `-server.grpc.network` (renamed from `-server.grpc-listen-network`) * `-server.grpc.conn-limit` (renamed from `-server.grpc-conn-limit`) * `-server.grpc.max-recv-msg-size-bytes` (renamed from `-server.grpc-max-recv-msg-size-bytes`) * `-server.grpc.max-send-msg-size-bytes` (renamed from `-server.grpc-max-send-msg-size-bytes`) * `-server.grpc.max-concurrent-streams` (renamed from `-server.grpc-max-concurrent-streams`) The following flags have been REMOVED: * `-log.level` (use `server.log_level` in the YAML instead) * `-log.format` (use `server.log_format` in the YAML instead) * `-server.http-tls-cert-path` (use `server.http_tls_config` in the YAML instead) * `-server.http-tls-key-path` (use `server.http_tls_config` in the YAML instead) * `-server.http-tls-client-auth` (use `server.http_tls_config` in the YAML instead) * `-server.http-tls-ca-path` (use `server.http_tls_config` in the YAML instead) * `-server.grpc-tls-cert-path` (use `server.grpc_tls_config` in the YAML instead) * `-server.grpc-tls-key-path` (use `server.grpc_tls_config` in the YAML instead) * `-server.grpc-tls-client-auth` (use `server.grpc_tls_config` in the YAML instead) * `-server.grpc-tls-ca-path` (use `server.grpc_tls_config` in the YAML instead) * `-server.http-listen-address` (use the `-server.http.address` flag instead) * `-server.http-listen-port` (use the `-server.http.address` flag instead) * `-server.grpc-listen-address` (use the `-server.grpc.address` flag instead) * `-server.grpc-listen-port` (use the `-server.grpc.address` flag instead) * `-server.path-prefix` (usage of this flag was never supported and would cause issues) `-server.path-prefix` and the equivalent YAML field `server.path_prefix` have been removed. Setting this option would lead to undefined behavior for various parts of the agent. Being able to prepend an API prefix may return in the future if it is requested. Flags not listed here are unchanged. Other changes ------------- All jsonnet and config file examples no longer use the deprecated field names. * pkg/server: make Test_tlsListener resilient to different environments * pkg/integrations: fix setting self-scraping host/port * pkg/server: remove outdated TODOs TODOs about documentation have been "documented" in the original commit message, and updating user documentation is being tracked externally on the PR. * pkg/server: fix help text for -server.log.source-ips.regex * cmd/agent: fix Windows compilation error * pkg/server: loosen test restriction on checking for reused port error * document changes * packaging/windows: fix arguments passed to NSIS * CHANGELOG: add missing authorship * pkg/server: split server.New implementation into multiple functions * pkg/server: add error checks back
Missing ">" in shortcode is breaking builds in /website repo
…ana-cold-storage#1480) * crow: make sure that default settings are applied to crow * explicitly set HTTP address in crow lib * tanka/v2: allow passing in custom flags and override exposed http port * tanka/v2: shift functions around * example/k3d/smoke: remove redundant argument
…les (grafana-cold-storage#1479) * packaging: fix issue where .debs and .rpms always overwrote config files * drone: use docker for running test-packages * drone: use docker as a base image for test-packages * i'm tired of writing commit messages every time i update drone * try the test-packages job on GitHub Actions
…following docs grafana-easystart-app/integrations-management/integrations/kubernetes :) (grafana-cold-storage#1484)
…#1481) * server: Change default listen addresses to 127.0.0.1:12345 / 127.0.0.1:12346 This commit changes the default listen addresses to be 127.0.0.1:12345 (for HTTP) and 127.0.0.1:12346 (for gRPC). This makes listening on all interfaces opt-in rather than opt-out, avoiding accidental overexposure of access. Closes grafana-cold-storage#852. Additionally, the `-reload-addr` and `-reload-port` flags have been removed as a follow up to grafana-cold-storage#1476. Now that the HTTP and gRPC server are static for the lifetime of the application, it is impossible for the user to change their configuration file in such a way to cause it to shut down while performing a reload. This means that the `-reload-addr` and `-reload-port` no longer have a use and can be removed safely. * fix extra references to old flags / port defaults * config: fix test * fix test assertion for reload address change
b0c1e45 accidentally removed setting `-log.level` in the smoke binary as part of removing the deprecated `-log.level` field from agent code.
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 1 to 3. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v1...v3) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* rfc: new metrics subsystem * add links to issues * remove sections describing dev branch workflow We have since stopped using dev branches, and recommend merging directly to main as long as PRs are tested and are not known to produce a non-functional or releasable build. * remove reference to deleted per-target sharding prototype
rfratto
requested review from
gaantunes,
joe-elliott,
mapno and
rgeyer
as code owners
March 15, 2022 13:37
|
|
github-actions
Bot
added
the
frozen-due-to-age
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.