Code to support dynamic configuration.

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 - [ENHANCEMENT] Go 1.17 is now used for all builds of the Agent. (@tpaschalis)
 
+- [ENHANCEMENT] Introduce EXPERIMENTAL support for dynamic configuration. (@mattdurham)
+
 - [BUGFIX] Fixed issue where Grafana Agent may panic if there is a very large
   WAL loading while old WALs are being deleted or the `/agent/api/v1/targets`
   endpoint is called. (@tpaschalis)

cmd/agentctl/main.go

@@ -10,6 +10,8 @@ import (
 	"strings"
 	"time"
 
+	"gopkg.in/yaml.v2"
+
 	// Adds version information
 	_ "github.com/grafana/agent/pkg/build"
 	"github.com/grafana/agent/pkg/client/grafanacloud"
@@ -60,6 +62,7 @@ func main() {
 		samplesCmd(),
 		operatorDetachCmd(),
 		cloudConfigCmd(),
+		templateDryRunCmd(),
 	)
 
 	_ = cmd.Execute()
@@ -474,6 +477,41 @@ config that may be used with this agent.`,
 	return cmd
 }
 
+func templateDryRunCmd() *cobra.Command {
+
+	cmd := &cobra.Command{
+		Use:   "template-parse [directory]",
+		Short: "dry run dynamic configuration",
+		Long:  `This will load the dynamic configuration, load configs, run templates and then output the full config as yaml`,
+		Args:  cobra.ExactArgs(1),
+
+		RunE: func(_ *cobra.Command, args []string) error {
+			cmf, err := config.NewDynamicLoader()
+			if err != nil {
+				return err
+			}
+			c := &config.Config{}
+			err = cmf.LoadConfigByPath(args[0])
+			if err != nil {
+				return err
+			}
+			err = cmf.ProcessConfigs(c)
+			if err != nil {
+				return fmt.Errorf("error processing config templates %s", err)
+			}
+
+			outBytes, err := yaml.Marshal(c)
+			if err != nil {
+				return err
+			}
+			fmt.Println(string(outBytes))
+			return nil
+		},
+	}
+
+	return cmd
+}
+
 func must(err error) {
 	if err != nil {
 		panic(err)

docs/configuration/_index.md

@@ -127,3 +127,8 @@ The following flags will configure basic auth for requests made to HTTP/S remote
 - `-config.url.basic-auth-password-file <file>`: path to a file containing the basic auth password
 
 Note that this beta feature is subject to change in future releases.
+
+## Dynamic Configuration (Experimental)
+
+Dynamic Configuration is an experimental feature that allows loading configurations from a variety of sources and 
+running gomplate templates against the found file. Details can be found in the [dynamic configuration config]({{< relref "./dynamic-config.md" >}})

docs/cookbook/dynamic-configuration/01_Basics/01_Structure.md

@@ -2,7 +2,7 @@
 
 Dynamic Configuration uses a series of files to load templates. This example will show how they all combine together. Running the below command will combine all the templates into the final.yml. Any failure while loading the config will revert to the original config, or if this is the initial load Grafana Agent will quit. 
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/01_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/01_config.yml`
 
 ## Dynamic Configuration
 

docs/cookbook/dynamic-configuration/01_Basics/01_assets/agent-1.yml

@@ -12,6 +12,8 @@ metrics:
           password: secretpassword
 integrations:
   node_exporter:
-    enabled: true
+    autoscrape:
+      enable: true
   agent:
-    enabled: true
+    autoscrape:
+      enable: true

docs/cookbook/dynamic-configuration/01_Basics/01_config.yml

@@ -1,2 +1,2 @@
 template_paths:
-  - "/etc/grafana/01_assets"
+  - "file:///etc/grafana/01_assets"

docs/cookbook/dynamic-configuration/01_Basics/02_Instances.md

@@ -3,7 +3,7 @@
 Dynamic configuration allows multiple prometheus instances to be loaded with a parent metric. This uses
 the same agent-1 and server-1 yml from 01.
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/02_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/02_config.yml`
 
 ## Dynamic Configuration
 

docs/cookbook/dynamic-configuration/01_Basics/02_assets/agent-1.yml

@@ -11,7 +11,5 @@ metrics:
           username: 12345
           password: secretpassword
 integrations:
-  node_exporter:
-    enabled: true
-  agent:
-    enabled: true
+  node_exporter: {}
+  agent: {}

docs/cookbook/dynamic-configuration/01_Basics/03_Integrations.md

@@ -2,7 +2,7 @@
 
 Dynamic configuration requires the use of `integrations-next` feature flag, to allow arrays of integrations. In this we will load integrations of various types. This is all built on the previous examples.
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/03_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/03_config.yml`
 
 ## Dynamic Configuration
 

docs/cookbook/dynamic-configuration/01_Basics/03_assets/agent-1.yml

@@ -11,7 +11,5 @@ metrics:
           username: 12345
           password: secretpassword
 integrations:
-  node_exporter:
-    enabled: true
-  agent:
-    enabled: true
+  node_exporter: {}
+  agent: {}

docs/cookbook/dynamic-configuration/01_Basics/04_Logs_and_Traces.md

@@ -2,7 +2,7 @@
 
 Logs and Traces can also be templated. This is built ontop of the previous examples.
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/04_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/04_config.yml`
 
 ## Dynamic Configuration
 

docs/cookbook/dynamic-configuration/01_Basics/04_assets/agent-1.yml

@@ -11,7 +11,5 @@ metrics:
           username: 12345
           password: secretpassword
 integrations:
-  node_exporter:
-    enabled: true
-  agent:
-    enabled: true
+  node_exporter: {}
+  agent: {}

docs/cookbook/dynamic-configuration/02_Templates/01_Looping.md

@@ -2,7 +2,7 @@
 
 The templating is based on the excellent [gomplate](https://docs.gomplate.ca/) library. Currently using a custom fork to allow loading gomplate as a library in addition to some new commands. This will NOT try to cover the full range of gomplate, would recommend reading the documentation for full knowledge. 
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/01_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/01_config.yml`
 
 ## Looping
 

docs/cookbook/dynamic-configuration/02_Templates/02_Datasources.md

@@ -2,7 +2,7 @@
 
 Datasources are a powerful concept in gomplate. They allow you to reach out to other files, systems and resources to pull data. 
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/02_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/02_config.yml`
 
 
 ## Config

docs/cookbook/dynamic-configuration/02_Templates/03_Datasource_and_Objects.md

@@ -2,7 +2,7 @@
 
 Datasources can also access objects.
 
-`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse /etc/grafana/03_config.yml`
+`docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/03_config.yml`
 
 
 ## Config