grafana-cold-storage · mattdurham · Feb 25, 2022 · Feb 25, 2022
diff --git a/docs/README.md b/docs/README.md
@@ -1,21 +1,10 @@
-+++
-draft = "True"
-+++
+# Grafana Agent Documentation
 
-<p align="center"><img src="assets/logo_and_name.png" alt="Grafana Agent logo"></p>
+This directory contains documentation for Grafana Agent. It is split into two
+parts:
 
-Grafana Agent is an telemetry collector for sending metrics, logs,
-and trace data to the opinionated Grafana observability stack. It works best
-with:
-
-* [Grafana Cloud](https://grafana.com/products/cloud/)
-* [Grafana Enterprise Stack](https://grafana.com/products/enterprise/)
-* OSS deployments of [Grafana Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), [Cortex](https://cortexmetrics.io/), and [Grafana Tempo](https://grafana.com/oss/tempo/)
-
-
-- Grafana Agent uses less memory on average than Prometheus – by doing less
-  (only focusing on `remote_write`-related functionality).
-- Grafana Agent allows for deploying multiple instances of the Agent in a
-  cluster and only scraping metrics from targets that are running on the same host.
-  This allows for distributing memory requirements across the cluster
-  rather than pressurizing a single node.
+* `user/`: Documentation for users. This directory is hosted on
+  [grafana.com/docs/agent](https://grafana.com/docs/agent/latest/) and we
+  recommend interacting with it there instead of viewing the markdown on
+  Github.
+* `rfcs/`: RFCs for proposals relating to Grafana Agent.
diff --git a/docs/cookbook/dynamic-configuration/README.md b/docs/cookbook/dynamic-configuration/README.md
diff --git a/docs/configuration/dynamic-config.md → docs/user/configuration/dynamic-config.md b/docs/configuration/dynamic-config.md → docs/user/configuration/dynamic-config.md
diff --git a/...c-configuration/01_Basics/01_Structure.md → ...c-configuration/01_Basics/01_Structure.md b/...c-configuration/01_Basics/01_Structure.md → ...c-configuration/01_Basics/01_Structure.md
@@ -6,7 +6,7 @@ Dynamic Configuration uses a series of files to load templates. This example wil
 
 ## Dynamic Configuration
 
-[config.yml](./01_config.yml)
+[config.yml](01_config.yml)
 
 ```yaml
 template_paths:
@@ -19,7 +19,7 @@ Tells the Grafana Agent where to load files from. It is important to note that d
 
 Dynamic Configuration will find the first file matching pattern `agent-*.yml` and load that as the base. You can only have one agent template. If multiple matching templates are found then the configuration will fail to load. 
 
-[agent-1.yml](./01_assets/agent-1.yml)
+[agent-1.yml](01_assets/agent-1.yml)
 
 ```yaml
 server:
@@ -49,7 +49,7 @@ is a configuration block.
 
 You can only have 1 server template.
 
-[server-1.yml](./01_assets/server-1.yml)
+[server-1.yml](01_assets/server-1.yml)
 
 
 ```yaml
@@ -59,6 +59,6 @@ log_level: info
 
 ## Final
 
-[final.yml](./01_assets/final.yml)
+[final.yml](01_assets/final.yml)
 
 In the above example the `log_level: debug` block will be replaced with `log_level: info` from the server-1.yml
diff --git a/...iguration/01_Basics/01_assets/agent-1.yml → ...iguration/01_Basics/01_assets/agent-1.yml b/...iguration/01_Basics/01_assets/agent-1.yml → ...iguration/01_Basics/01_assets/agent-1.yml
diff --git a/...nfiguration/01_Basics/01_assets/final.yml → ...nfiguration/01_Basics/01_assets/final.yml b/...nfiguration/01_Basics/01_assets/final.yml → ...nfiguration/01_Basics/01_assets/final.yml
diff --git a/...guration/01_Basics/01_assets/server-1.yml → ...guration/01_Basics/01_assets/server-1.yml b/...guration/01_Basics/01_assets/server-1.yml → ...guration/01_Basics/01_assets/server-1.yml
diff --git a/...mic-configuration/01_Basics/01_config.yml → ...mic-configuration/01_Basics/01_config.yml b/...mic-configuration/01_Basics/01_config.yml → ...mic-configuration/01_Basics/01_config.yml
diff --git a/...c-configuration/01_Basics/02_Instances.md → ...c-configuration/01_Basics/02_Instances.md b/...c-configuration/01_Basics/02_Instances.md → ...c-configuration/01_Basics/02_Instances.md
@@ -7,15 +7,15 @@ the same agent-1 and server-1 yml from 01.
 
 ## Dynamic Configuration
 
-[config.yml](./02_config.yml)
+[config.yml](02_config.yml)
 
 Tells the Grafana Agent where to load files from.
 
 ## Metrics
 
 Dynamic Configuration will find the first file matching pattern `metrics-*.yml` and load that as the base. You can only have one metrics template.
 
-[metrics-1.yml](./02_assets/metrics-1.yml)
+[metrics-1.yml](02_assets/metrics-1.yml)
 
 ```yaml
 configs:
@@ -30,7 +30,7 @@ wal_directory: /tmp/grafana-agent-wal
 
 You can have any number of metrics_instances and they are added to any existing metrics instances defined previously.
 
-[metrics_instances-1.yml](./02_assets/metrics_instances-1.yml)
+[metrics_instances-1.yml](02_assets/metrics_instances-1.yml)
 
 ```yaml
 name: instance1
@@ -41,7 +41,7 @@ scrape_configs:
           - localhost:4000
 ```
 
-[metrics_instances-2.yml](./02_assets/metrics_instances-2.yml)
+[metrics_instances-2.yml](02_assets/metrics_instances-2.yml)
 
 ```yaml
 name: instance2
@@ -54,7 +54,7 @@ scrape_configs:
 
 ## Final
 
-[final.yml](./01_assets/final.yml)
+[final.yml](01_assets/final.yml)
 
 In the above you will see the `final.yml` includes all the instance configurations
 - default

diff --git a/...iguration/01_Basics/02_assets/agent-1.yml → ...iguration/01_Basics/02_assets/agent-1.yml b/...iguration/01_Basics/02_assets/agent-1.yml → ...iguration/01_Basics/02_assets/agent-1.yml
diff --git a/...nfiguration/01_Basics/02_assets/final.yml → ...nfiguration/01_Basics/02_assets/final.yml b/...nfiguration/01_Basics/02_assets/final.yml → ...nfiguration/01_Basics/02_assets/final.yml
diff --git a/...uration/01_Basics/02_assets/metrics-1.yml → ...uration/01_Basics/02_assets/metrics-1.yml b/...uration/01_Basics/02_assets/metrics-1.yml → ...uration/01_Basics/02_assets/metrics-1.yml
diff --git a/..._Basics/02_assets/metrics_instances-1.yml → ..._Basics/02_assets/metrics_instances-1.yml b/..._Basics/02_assets/metrics_instances-1.yml → ..._Basics/02_assets/metrics_instances-1.yml
diff --git a/..._Basics/02_assets/metrics_instances-2.yml → ..._Basics/02_assets/metrics_instances-2.yml b/..._Basics/02_assets/metrics_instances-2.yml → ..._Basics/02_assets/metrics_instances-2.yml
diff --git a/...guration/01_Basics/02_assets/server-1.yml → ...guration/01_Basics/02_assets/server-1.yml b/...guration/01_Basics/02_assets/server-1.yml → ...guration/01_Basics/02_assets/server-1.yml
diff --git a/...mic-configuration/01_Basics/02_config.yml → ...mic-configuration/01_Basics/02_config.yml b/...mic-configuration/01_Basics/02_config.yml → ...mic-configuration/01_Basics/02_config.yml
diff --git a/...onfiguration/01_Basics/03_Integrations.md → ...onfiguration/01_Basics/03_Integrations.md b/...onfiguration/01_Basics/03_Integrations.md → ...onfiguration/01_Basics/03_Integrations.md
@@ -6,23 +6,23 @@ Dynamic configuration requires the use of `integrations-next` feature flag, to a
 
 ## Dynamic Configuration
 
-[config.yml](./03_config.yml)
+[config.yml](03_config.yml)
 
 Tells the Grafana Agent where to load files from.
 
 ## Integrations
 
 Integrations are loaded from files matching `integrations-*.yml` and are combined together. You can declare for example multiple sets of `redis_exporter_configs` across several files. 
 
-[integrations-node.yml](./03_assets/integrations-node.yml)
+[integrations-node.yml](03_assets/integrations-node.yml)
 
 Note: You do NOT have to name the above file `integrations-node.yml` with `node`, `integrations-1.yml` would work the same. The name does NOT determine the type of integrations a template can contain and a template can contain integrations of different types.  
 
 ```yaml
 node_exporter: {}
 ```
 
-[integrations-redis.yml](./03_assets/integrations-redis.yml)
+[integrations-redis.yml](03_assets/integrations-redis.yml)
 
 ```yaml
 redis_exporter_configs:
@@ -37,7 +37,7 @@ redis_exporter_configs:
 
 ## Final
 
-[final.yml](./03_assets/final.yml)
+[final.yml](03_assets/final.yml)
 
 The final result should have 3 integrations enabled, 1 node_exporter and 2 redis_exporters.
 
diff --git a/...iguration/01_Basics/03_assets/agent-1.yml → ...iguration/01_Basics/03_assets/agent-1.yml b/...iguration/01_Basics/03_assets/agent-1.yml → ...iguration/01_Basics/03_assets/agent-1.yml
diff --git a/...nfiguration/01_Basics/03_assets/final.yml → ...nfiguration/01_Basics/03_assets/final.yml b/...nfiguration/01_Basics/03_assets/final.yml → ...nfiguration/01_Basics/03_assets/final.yml
diff --git a/...01_Basics/03_assets/integrations-node.yml → ...01_Basics/03_assets/integrations-node.yml b/...01_Basics/03_assets/integrations-node.yml → ...01_Basics/03_assets/integrations-node.yml
diff --git a/...1_Basics/03_assets/integrations-redis.yml → ...1_Basics/03_assets/integrations-redis.yml b/...1_Basics/03_assets/integrations-redis.yml → ...1_Basics/03_assets/integrations-redis.yml
diff --git a/...uration/01_Basics/03_assets/metrics-1.yml → ...uration/01_Basics/03_assets/metrics-1.yml b/...uration/01_Basics/03_assets/metrics-1.yml → ...uration/01_Basics/03_assets/metrics-1.yml
diff --git a/..._Basics/03_assets/metrics_instances-1.yml → ..._Basics/03_assets/metrics_instances-1.yml b/..._Basics/03_assets/metrics_instances-1.yml → ..._Basics/03_assets/metrics_instances-1.yml
diff --git a/..._Basics/03_assets/metrics_instances-2.yml → ..._Basics/03_assets/metrics_instances-2.yml b/..._Basics/03_assets/metrics_instances-2.yml → ..._Basics/03_assets/metrics_instances-2.yml
diff --git a/...guration/01_Basics/03_assets/server-1.yml → ...guration/01_Basics/03_assets/server-1.yml b/...guration/01_Basics/03_assets/server-1.yml → ...guration/01_Basics/03_assets/server-1.yml
diff --git a/...mic-configuration/01_Basics/03_config.yml → ...mic-configuration/01_Basics/03_config.yml b/...mic-configuration/01_Basics/03_config.yml → ...mic-configuration/01_Basics/03_config.yml
diff --git a/...iguration/01_Basics/04_Logs_and_Traces.md → ...iguration/01_Basics/04_Logs_and_Traces.md b/...iguration/01_Basics/04_Logs_and_Traces.md → ...iguration/01_Basics/04_Logs_and_Traces.md
@@ -6,15 +6,15 @@ Logs and Traces can also be templated. This is built ontop of the previous examp
 
 ## Dynamic Configuration
 
-[config.yml](./04_config.yml)
+[config.yml](04_config.yml)
 
 Tells the Grafana Agent where to load files from.
 
 ## Logs
 
 Logs are loaded from a template matching `logs-*.yml`. There can ONLY be 1 template loaded
 
-[logs-1.yml](./04_assets/logs-1.yml)
+[logs-1.yml](04_assets/logs-1.yml)
 
 ```yaml
 configs:
@@ -29,7 +29,7 @@ configs:
             expression: '\\temp\\Logs\\(?P<log_app>.+?)\\'
 ```
 
-[traces.yml](./04_assets/traces-1.yml)
+[traces.yml](04_assets/traces-1.yml)
 
 ```yaml
 configs:
@@ -42,5 +42,5 @@ configs:
 
 ## Final
 
-[final.yml](./04_assets/final.yml)
+[final.yml](04_assets/final.yml)
 
diff --git a/...iguration/01_Basics/04_assets/agent-1.yml → ...iguration/01_Basics/04_assets/agent-1.yml b/...iguration/01_Basics/04_assets/agent-1.yml → ...iguration/01_Basics/04_assets/agent-1.yml
diff --git a/...nfiguration/01_Basics/04_assets/final.yml → ...nfiguration/01_Basics/04_assets/final.yml b/...nfiguration/01_Basics/04_assets/final.yml → ...nfiguration/01_Basics/04_assets/final.yml
diff --git a/...01_Basics/04_assets/integrations-node.yml → ...01_Basics/04_assets/integrations-node.yml b/...01_Basics/04_assets/integrations-node.yml → ...01_Basics/04_assets/integrations-node.yml
diff --git a/...1_Basics/04_assets/integrations-redis.yml → ...1_Basics/04_assets/integrations-redis.yml b/...1_Basics/04_assets/integrations-redis.yml → ...1_Basics/04_assets/integrations-redis.yml
diff --git a/...figuration/01_Basics/04_assets/logs-1.yml → ...figuration/01_Basics/04_assets/logs-1.yml b/...figuration/01_Basics/04_assets/logs-1.yml → ...figuration/01_Basics/04_assets/logs-1.yml
diff --git a/...uration/01_Basics/04_assets/metrics-1.yml → ...uration/01_Basics/04_assets/metrics-1.yml b/...uration/01_Basics/04_assets/metrics-1.yml → ...uration/01_Basics/04_assets/metrics-1.yml
diff --git a/..._Basics/04_assets/metrics_instances-1.yml → ..._Basics/04_assets/metrics_instances-1.yml b/..._Basics/04_assets/metrics_instances-1.yml → ..._Basics/04_assets/metrics_instances-1.yml
diff --git a/..._Basics/04_assets/metrics_instances-2.yml → ..._Basics/04_assets/metrics_instances-2.yml b/..._Basics/04_assets/metrics_instances-2.yml → ..._Basics/04_assets/metrics_instances-2.yml
diff --git a/...guration/01_Basics/04_assets/server-1.yml → ...guration/01_Basics/04_assets/server-1.yml b/...guration/01_Basics/04_assets/server-1.yml → ...guration/01_Basics/04_assets/server-1.yml
diff --git a/...guration/01_Basics/04_assets/traces-1.yml → ...guration/01_Basics/04_assets/traces-1.yml b/...guration/01_Basics/04_assets/traces-1.yml → ...guration/01_Basics/04_assets/traces-1.yml
diff --git a/...mic-configuration/01_Basics/04_config.yml → ...mic-configuration/01_Basics/04_config.yml b/...mic-configuration/01_Basics/04_config.yml → ...mic-configuration/01_Basics/04_config.yml
diff --git a/...-configuration/02_Templates/01_Looping.md → ...-configuration/02_Templates/01_Looping.md b/...-configuration/02_Templates/01_Looping.md → ...-configuration/02_Templates/01_Looping.md
@@ -6,7 +6,7 @@ The templating is based on the excellent [gomplate](https://docs.gomplate.ca/) l
 
 ## Looping
 
-[agent-1.yml](./01_assets/agent-1.yml)
+[agent-1.yml](01_assets/agent-1.yml)
 
 ```yaml
 server:
@@ -32,7 +32,7 @@ The templating engine uses directives that are wrapped in `{{ command }}`, in th
 
 ## Final
 
-[final.yml](./01_assets/final.yml)
+[final.yml](01_assets/final.yml)
 
 The final.yml contains 4 prometheus configs
 

diff --git a/...ration/02_Templates/01_assets/agent-1.yml → ...ration/02_Templates/01_assets/agent-1.yml b/...ration/02_Templates/01_assets/agent-1.yml → ...ration/02_Templates/01_assets/agent-1.yml
diff --git a/...guration/02_Templates/01_assets/final.yml → ...guration/02_Templates/01_assets/final.yml b/...guration/02_Templates/01_assets/final.yml → ...guration/02_Templates/01_assets/final.yml
diff --git a/...-configuration/02_Templates/01_config.yml → ...-configuration/02_Templates/01_config.yml b/...-configuration/02_Templates/01_config.yml → ...-configuration/02_Templates/01_config.yml
diff --git a/...figuration/02_Templates/02_Datasources.md → ...figuration/02_Templates/02_Datasources.md b/...figuration/02_Templates/02_Datasources.md → ...figuration/02_Templates/02_Datasources.md
@@ -7,7 +7,7 @@ Datasources are a powerful concept in gomplate. They allow you to reach out to o
 
 ## Config
 
-The [config.yml](./02_config.yml) adds a new field `sources`. Sources can be any number of things defined in the gomplate [datasources](https://docs.gomplate.ca/datasources/) documentation. In this example using fruit.
+The [config.yml](02_config.yml) adds a new field `sources`. Sources can be any number of things defined in the gomplate [datasources](https://docs.gomplate.ca/datasources/) documentation. In this example using fruit.
 
 ```yaml
 template_paths:
@@ -17,15 +17,15 @@ datasources:
     url: "file://etc/grafana/01_assets/fruit.json"
 ```
 
-[fruit.json](./02_assets/fruit.json)
+[fruit.json](02_assets/fruit.json)
 
 ```json
 ["mango","peach","orange"]
 ```
 
 ## Usage
 
-[agent-1.yml](./02_assets/agent-1.yml)
+[agent-1.yml](02_assets/agent-1.yml)
 
 ```yaml
 server:

diff --git a/...ration/02_Templates/02_assets/agent-1.yml → ...ration/02_Templates/02_assets/agent-1.yml b/...ration/02_Templates/02_assets/agent-1.yml → ...ration/02_Templates/02_assets/agent-1.yml
diff --git a/...guration/02_Templates/02_assets/final.yml → ...guration/02_Templates/02_assets/final.yml b/...guration/02_Templates/02_assets/final.yml → ...guration/02_Templates/02_assets/final.yml
diff --git a/...uration/02_Templates/02_assets/fruit.json → ...uration/02_Templates/02_assets/fruit.json b/...uration/02_Templates/02_assets/fruit.json → ...uration/02_Templates/02_assets/fruit.json
diff --git a/...-configuration/02_Templates/02_config.yml → ...-configuration/02_Templates/02_config.yml b/...-configuration/02_Templates/02_config.yml → ...-configuration/02_Templates/02_config.yml
diff --git a/...02_Templates/03_Datasource_and_Objects.md → ...02_Templates/03_Datasource_and_Objects.md b/...02_Templates/03_Datasource_and_Objects.md → ...02_Templates/03_Datasource_and_Objects.md
@@ -7,7 +7,7 @@ Datasources can also access objects.
 
 ## Config
 
-The [config.yml](./02_config.yml) adds a new field `sources`. Sources can be any number of things defined in the gomplate [datasources](https://docs.gomplate.ca/datasources/) documentation. In this example using fruit.
+The [config.yml](02_config.yml) adds a new field `sources`. Sources can be any number of things defined in the gomplate [datasources](https://docs.gomplate.ca/datasources/) documentation. In this example using fruit.
 
 ```yaml
 template_paths:
@@ -17,7 +17,7 @@ datasources:
     url: "file:///etc/grafana/03_assets/computers.json"
 ```
 
-[computers.json](./03_assets/computers.json)
+[computers.json](03_assets/computers.json)
 
 ```json
 [
@@ -42,7 +42,7 @@ datasources:
 
 ## Usage
 
-[agent-1.yml](./02_assets/agent-1.yml)
+[agent-1.yml](02_assets/agent-1.yml)
 
 ```yaml
 server:

diff --git a/...ration/02_Templates/03_assets/agent-1.yml → ...ration/02_Templates/03_assets/agent-1.yml b/...ration/02_Templates/03_assets/agent-1.yml → ...ration/02_Templates/03_assets/agent-1.yml
diff --git a/...ion/02_Templates/03_assets/computers.json → ...ion/02_Templates/03_assets/computers.json b/...ion/02_Templates/03_assets/computers.json → ...ion/02_Templates/03_assets/computers.json
diff --git a/...guration/02_Templates/03_assets/final.yml → ...guration/02_Templates/03_assets/final.yml b/...guration/02_Templates/03_assets/final.yml → ...guration/02_Templates/03_assets/final.yml
diff --git a/...-configuration/02_Templates/03_config.yml → ...-configuration/02_Templates/03_config.yml b/...-configuration/02_Templates/03_config.yml → ...-configuration/02_Templates/03_config.yml
diff --git a/...uration/03_Advanced_Datasources/01_AWS.md → ...uration/03_Advanced_Datasources/01_AWS.md b/...uration/03_Advanced_Datasources/01_AWS.md → ...uration/03_Advanced_Datasources/01_AWS.md
@@ -6,7 +6,7 @@ Unfortunately there is not a specific docker command but generic examples are be
 
 ## Looping
 
-[agent-1.yml](./01_assets/agent-1.yml)
+[agent-1.yml](01_assets/agent-1.yml)
 
 ```yaml
 server:
@@ -36,4 +36,4 @@ The `aws.EC2Query` command is a new command added for Grafana Agent and takes a
 
 ## Final
 
-[final.yml](./01_assets/final.yml)
+[final.yml](01_assets/final.yml)
diff --git a/...dvanced_Datasources/01_assets/agent-1.yml → ...dvanced_Datasources/01_assets/agent-1.yml b/...dvanced_Datasources/01_assets/agent-1.yml → ...dvanced_Datasources/01_assets/agent-1.yml
diff --git a/..._Advanced_Datasources/01_assets/final.yml → ..._Advanced_Datasources/01_assets/final.yml b/..._Advanced_Datasources/01_assets/final.yml → ..._Advanced_Datasources/01_assets/final.yml
diff --git a/...ion/03_Advanced_Datasources/01_config.yml → ...ion/03_Advanced_Datasources/01_config.yml b/...ion/03_Advanced_Datasources/01_config.yml → ...ion/03_Advanced_Datasources/01_config.yml
diff --git a/docs/user/cookbook/dynamic-configuration/README.md b/docs/user/cookbook/dynamic-configuration/README.md
@@ -0,0 +1,19 @@
+# Dynamic Configuration Cookbook
+
+The purpose of the cookbook is to guide you through common scenarios of using dynamic configuration. Each folder contains increasingly more complex use cases, but feel free to jump in wherever you feel appropriate.
+
+## Basics
+
+[Basics](01_Basics) covers
+- [Structure](01_Basics/01_Structure.md) of how agent and server templates are loaded
+- [Instances](01_Basics/02_Instances.md) of metrics and metrics instances are loaded
+- [Integrations](01_Basics/03_Integrations.md) of how integrations are loaded
+- [Logs and Traces](01_Basics/04_Logs_and_Traces.md) of how traces and logs are loaded
+
+[Templates](02_Templates) covers
+- [Looping](02_Templates/01_Looping.md) covers basic command usage and simple loops
+- [Datasource](02_Templates/02_Datasources.md) covers usage of datasource which are external datastores you can use to pull in data
+- [Datasources and Objects](02_Templates/03_Datasource_and_Objects.md) covers the usage of complex json objects
+
+[Advanced Datasources](03_Advanced_Datasources) covers non file based datasources
+- [AWS](03_Advanced_Datasources/01_AWS.md) covers querying EC2 for instances