Merge of main into dev.servicegraphs

.drone/README.md

@@ -0,0 +1,8 @@
+When updating the drone.yml file the file must be re-signed using `make drone`. This is limited to Grafana employees for security reasons.
+
+Drone environment variables will need to be setup beforehand.
+
+```
+export DRONE_SERVER=<url>
+export DRONE_TOKEN=<token>
+```

.drone/drone.yml

@@ -90,6 +90,7 @@ trigger:
   ref:
     - refs/heads/main
     - refs/tags/v*
+    - refs/heads/dev.*
 
 steps:
   - name: Build Containers
@@ -153,7 +154,7 @@ steps:
       - apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release
       - curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
       - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
-      - apt-get update &&  apt-get install -y rubygems rpm nsis docker-ce docker-ce-cli containerd.io
+      - apt-get update &&  apt-get install -y rubygems rpm nsis docker-ce docker-ce-cli containerd.io gettext
       - docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD
       - gem install --no-document fpm
       - mkdir -p /usr/local/go/bin
@@ -173,6 +174,6 @@ volumes:
       path: /var/run/docker.sock
 ---
 kind: signature
-hmac: 45dd2046cb9f7392b2dfac3ace3c88966d48a9cce9828e64e39ddefb477b4dfb
+hmac: 1cab98e575942179c9a157058799b9d44119b1d6c170feefbeefe303eb44823d
 
 ...

.github/depcheck.yml

@@ -3,7 +3,7 @@
 go_modules:
   - github.com/cortexproject/cortex
   - github.com/grafana/loki
-  - github.com/justwatchcom/elasticsearch_exporter
+  - github.com/prometheus-community/elasticsearch_exporter
   - github.com/oliver006/redis_exporter
   - github.com/prometheus/consul_exporter
   - github.com/prometheus/memcached_exporter
@@ -21,8 +21,8 @@ go_modules:
 github_repos:
   - github.com/google/dnsmasq_exporter v0.2.0
   - github.com/ncabatoff/process-exporter v0.7.5
-  - github.com/prometheus/mysqld_exporter v0.12.1
-  - github.com/wrouesnel/postgres_exporter v0.8.0
+  - github.com/prometheus/mysqld_exporter v0.13.0
+  - github.com/prometheus-community/postgres_exporter v0.10.0
   - project: github.com/prometheus/prometheus
     version: v2.27.0
     # Ignore release candidates

.github/workflows/markdown.links.config.json

@@ -2,6 +2,9 @@
   "ignorePatterns": [
     {
       "pattern": "^http://prometheus.k3d.localhost:30080"
+    },
+    {
+      "pattern": "^http://prometheus.k3d.localhost:50080"
     }
   ]
 }

.github/workflows/publish.yml

@@ -0,0 +1,29 @@
+name: publish_docs
+
+on:
+  push:
+    branches:
+    - main
+    paths:
+    - 'docs/**'
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v1
+    - run: git clone --single-branch --no-tags --depth 1 -b master https://grafanabot:${{ secrets.GH_BOT_ACCESS_TOKEN }}@github.com/grafana/website-sync ./.github/actions/website-sync
+    - name: publish-to-git
+      uses: ./.github/actions/website-sync
+      id: publish
+      with:
+        repository: grafana/website
+        branch: master
+        host: github.com
+        github_pat: '${{ secrets.GH_BOT_ACCESS_TOKEN }}'
+        source_folder: docs
+        target_folder: content/docs/agent/latest
+    - shell: bash
+      run: |
+        test -n "${{ steps.publish.outputs.commit_hash }}"
+        test -n "${{ steps.publish.outputs.working_directory }}"

.gitignore

@@ -1,21 +1,20 @@
-.rpmdb/
-.idea
-.uptodate
-.published
-.pkg
-.cache
-.vscode
-vendor
-cover*.out
-vendor
+/.rpmdb/
+/.idea
+/.published
+/.pkg
+/.cache
+/.vscode
+/vendor
 
-cmd/agent/agent
-cmd/agentctl/agentctl
-cmd/agent-operator/agent-operator
-cmd/grafana-agent-crow/grafana-agent-crow
-dist/
-packaging/windows/LICENSE
-packaging/windows/agent-windows-amd64.exe
+/cmd/agent/agent
+/cmd/agentctl/agentctl
+/cmd/agent-operator/agent-operator
+/cmd/grafana-agent-crow/grafana-agent-crow
+/dist/
+/packaging/windows/LICENSE
+/packaging/windows/agent-windows-amd64.exe
 
 .DS_Store
 buildx-v*
+cover*.out
+.uptodate

CHANGELOG.md

@@ -1,10 +1,36 @@
 # Main (unreleased)
 
+- [FEATURE] Operator: The Grafana Agent Operator can now generate a Kubelet
+  service to allow a ServiceMonitor to collect Kubelet and cAdvisor metrics.
+  This requires passing a `--kubelet-service` flag to the Operator in
+  `namespace/name` format (like `kube-system/kubelet`). (@rfratto)
+
+- [FEATURE] Service graphs processor (@mapno)
+
+- [ENHANCEMENT] Updated mysqld_exporter to v0.13.0 (@gaantunes)
+
+- [ENHANCEMENT] Updated postgres_exporter to v0.10.0 (@gaantunes)
+
+- [ENHANCEMENT] Updated redis_exporter to v1.27.1 (@gaantunes)
+
+- [ENHANCEMENT] Updated memcached_exporter to v0.9.0 (@gaantunes)
+
+- [ENHANCEMENT] Updated statsd_exporter to v0.22.2 (@gaantunes)
+
+- [ENHANCEMENT] Updated elasticsearch_exporter to v1.2.1 (@gaantunes)
+
+- [ENHANCEMENT] Add remote write to silent Windows Installer  (@mattdurham)
+
+- [BUGFIX] Sanitize autologged Loki labels by replacing invalid characters with underscores (@mapno)
+
+# v0.19.0 (2021-09-29)
+
 This release has breaking changes. Please read [CHANGE] entries carefully and
 consult the
 [upgrade guide](https://github.com/grafana/agent/blob/main/docs/upgrade-guide/_index.md)
 for specific instructions.
 
+
 - [FEATURE] Added [Github exporter](https://github.com/infinityworks/github-exporter) integration. (@rgeyer)
 
 - [FEATURE] Add TLS config options for tempo `remote_write`s. (@mapno)
@@ -18,8 +44,6 @@ for specific instructions.
 - [FEATURE] Add `operator-detach` command to agentctl to allow zero-downtime
   upgrades when removing an Operator CRD. (@rfratto)
 
-- [FEATURE] Service graphs processor (@mapno)
-
 - [ENHANCEMENT] The Grafana Agent Operator will now default to deploying
   the matching release version of the Grafana Agent instead of v0.14.0.
   (@rfratto)
@@ -31,19 +55,23 @@ for specific instructions.
 - [ENHANCEMENT] Add HOSTNAME environment variable to service file to allow for expanding
   the $HOSTNAME variable in agent config.  (@dfrankel33)
 
+- [ENHANCEMENT] Update jsonnet-libs to 1.21 for Kubernetes 1.21+ compatability. (@MurzNN)
+
+- [ENHANCEMENT] Make method used to add k/v to spans in prom_sd processor
+  configurable. (@mapno)
+
 - [BUGFIX] Regex capture groups like `${1}` will now be kept intact when
   using `-config.expand-env`. (@rfratto)
 
 - [BUGFIX] The directory of the logs positions file will now properly be created
-  on startup for all instances.
+  on startup for all instances. (@rfratto)
 
 - [BUGFIX] The Linux system packages will now configure the grafana-agent user
   to be a member of the adm and systemd-journal groups. This will allow logs to
   read from journald and /var/log by default. (@rfratto)
 
-- [BUGFIX] Fix collecting filesystem metrics on Mac OS (darwin) in the `node_exporter` integration default config. (@eamonryan)
-
-- [BUGFIX] Fix info logging on windows. (@mattdurham)
+- [BUGFIX] Fix collecting filesystem metrics on Mac OS (darwin) in the
+  `node_exporter` integration default config. (@eamonryan)
 
 - [BUGFIX] Remove v0.0.0 flags during build with no explicit release tag (@mattdurham)
 
@@ -57,9 +85,9 @@ for specific instructions.
 
 - [BUGFIX] Fix yaml marshalling tag for cert_file in kafka exporter agent config. (@rgeyer)
 
-- [BUGFIX] Register missing metric for configstore consul request duration.
+- [BUGFIX] Fix warn-level logging of dropped targets. (@james-callahan)
 
-- [BUGFIX] Logs should contain a caller field with file and line numbers again (@kgeckhart)
+- [BUGFIX] Standardize scrape_interval to 1m in examples. (@mattdurham)
 
 - [CHANGE] Breaking change: reduced verbosity of tracing autologging
   by not logging `STATUS_CODE_UNSET` status codes. (@mapno)
@@ -71,12 +99,54 @@ for specific instructions.
   hyphen in the name to be consistent with how Kubernetes refers to resources.
   (@rfratto)
 
+- [CHANGE] Breaking change: `prom_instance` in the spanmetrics config is now
+  named `metrics_instance`. (@rfratto)
+
 - [DEPRECATION] The `loki` key at the root of the config file has been
   deprecated in favor of `logs`. `loki`-named fields in `automatic_logging`
   have been renamed accordinly: `loki_name` is now `logs_instance_name`,
   `loki_tag` is now `logs_instance_tag`, and `backend: loki` is now
   `backend: logs_instance`. (@rfratto)
 
+- [DEPRECATION] The `prometheus` key at the root of the config file has been
+  deprecated in favor of `metrics`. Flag names starting with `prometheus.` have
+  also been deprecated in favor of the same flags with the `metrics.` prefix.
+  Metrics prefixed with `agent_prometheus_` are now prefixed with
+  `agent_metrics_`. (@rfratto)
+
+- [DEPRECATION] The `tempo` key at the root of the config file has been
+  deprecated in favor of `traces`. (@mattdurham)
+
+# v0.18.4 (2021-09-14)
+
+- [BUGFIX] Fix info logging on windows. (@mattdurham)
+
+- [BUGFIX] Scraping service: Ensure that a reshard is scheduled every reshard
+  interval. (@rfratto)
+
+- [CHANGE] Add `agent_prometheus_configs_changed_total` metric to track instance
+  config events. (@rfratto)
+
+# v0.18.3 (2021-09-08)
+
+- [BUGFIX] Register missing metric for configstore consul request duration.
+  (@rfratto)
+
+- [BUGFIX] Logs should contain a caller field with file and line numbers again
+  (@kgeckhart)
+
+- [BUGFIX] In scraping service mode, the polling configuration refresh should
+  honor timeout. (@mattdurham)
+
+- [BUGFIX] In scraping service mode, the lifecycle reshard should happen using a
+  goroutine. (@mattdurham)
+
+- [BUGFIX] In scraping service mode, scraping service can deadlock when
+  reloading during join. (@mattdurham)
+
+- [BUGFIX] Scraping service: prevent more than one refresh from being queued at
+  a time. (@rfratto)
+
 # v0.18.2 (2021-08-12)
 
 - [BUGFIX] Honor the prefix and remove prefix from consul list results (@mattdurham)

CODEOWNERS

@@ -1 +1 @@
-/pkg/tempo/ @joe-elliott @mapno
+/pkg/traces/ @joe-elliott @mapno

MAINTAINERS.md

@@ -1,7 +1,7 @@
 Robert Fratto (<robert.fratto@grafana.com> / @rfratto) is the main/default maintainer, some parts of the codebase have other maintainers:
 
 * `pkg`
-  * `tempo`: Joe Elliott (<joe.elliott@grafana.com> / @joe-elliott), Mario Rodriguez (<mario.rodriguez@grafana.com>/ @mapno)
+  * `traces`: Joe Elliott (<joe.elliott@grafana.com> / @joe-elliott), Mario Rodriguez (<mario.rodriguez@grafana.com>/ @mapno)
 
 For the sake of brevity, not all subtrees are explicitly listed. Due to the
 size of this repository, the natural changes in focus of maintainers over time,

README.md

@@ -52,29 +52,9 @@ means that reliability of alerts are tied to the reliability of the remote syste
 and alerts will be delayed at least by the time it takes for samples to reach
 the remote system.
 
-## Roadmap
-
-- [x] Prometheus metrics
-- [x] A second clustering mode to solve sharding monitoring availability problems.
-- [x] Support for integrations (embedded exporters/automatic scrape configs)
-- [x] Promtail for Loki logs
-- [x] Tempo traces
-- [ ] `carbon-relay-ng` for Graphite metrics.
-- [ ] All-in-one installation script (metrics, logs, and traces)
-
 ## Getting Started
 
-The easiest way to get started with the Grafana Agent is to use the
-Kubernetes install scripts. The first script installs an Agent for collecting
-metrics, the second for collecting logs, and the third for collecting traces.
-Simply copy and paste the following lines in your terminal (requires `envsubst`
-(GNU gettext)):
-
-```
-NAMESPACE="default" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install.sh)" | kubectl apply -f -
-NAMESPACE="default" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-loki.sh)" | kubectl apply -f -
-NAMESPACE="default" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-tempo.sh)" | kubectl apply -f -
-```
+When using Kubernetes this [link](https://grafana.com/docs/grafana-cloud/quickstart/agent-k8s) offers the best guide. 
 
 Other installation methods can be found in our
 [Production](./production/README.md) documentation.
@@ -115,4 +95,8 @@ If you have any questions or feedback regarding the Grafana Agent:
   [Discussions page](https://github.com/grafana/agent/discussions).
 * [File an issue](https://github.com/grafana/agent/issues/new) for bugs, issues
   and feature suggestions.
+
+## Contributing
+
+Any contributions are welcome and details can be found [in our developers guide](./cmd/agent/DEVELOPERS.md).