-
Notifications
You must be signed in to change notification settings - Fork 2.1k
OCB init #14530
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
OCB init #14530
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
3f229af
add new ocb init command
dmathieu d290e2b
use path.Join to handle windows
dmathieu 06f91bb
fix relative path on windows
dmathieu ec3f7fd
Update cmd/builder/internal/command_init.go
dmathieu 0c1edf5
initialize go mod
dmathieu 521d409
ignore tmp folder from git
dmathieu b9833fa
add makefile and default collector configuration
dmathieu 78359d3
ensure the generated collector can run
dmathieu 4504e4a
wait for a minute
dmathieu c519edf
add changelog entry
dmathieu 2297284
go up to 2 minutes
dmathieu 26d1e41
add pr number
dmathieu f25e802
go up to 5 minutes
dmathieu 21e550a
move the e2e test to test.sh
dmathieu fe15d21
wait for 5 minutes for init
dmathieu 90be5f5
use compiled versions from the builder config
dmathieu 189489d
fix imports order
dmathieu d4411e7
Update cmd/builder/internal/command_init.go
dmathieu 40ac96f
add readme
dmathieu e840f1f
use flag to pass path in integration test
dmathieu 6f9a58a
properly name init test
dmathieu 2dfe3f4
Update cmd/builder/internal/init/templates/README.md.tmpl
dmathieu 420dbca
Update cmd/builder/internal/init/templates/README.md.tmpl
dmathieu 43c7ef6
Update cmd/builder/internal/init/templates/README.md.tmpl
dmathieu ab89f5d
Update cmd/builder/internal/init/templates/README.md.tmpl
dmathieu ec27481
Update cmd/builder/internal/init/templates/README.md.tmpl
dmathieu 8e17c3d
add link to writing new components
dmathieu 158e808
singular receiver and exporter
dmathieu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Use this changelog template to create an entry for release notes. | ||
|
|
||
| # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' | ||
| change_type: enhancement | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. receiver/otlp) | ||
| component: cmd/builder | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: Introduce new experimental `init` subcommand | ||
|
|
||
| # One or more tracking issues or pull requests related to the change | ||
| issues: [14530] | ||
|
|
||
| # (Optional) One or more lines of additional information to render under the primary note. | ||
| # These lines will be padded with 2 spaces and then inserted directly into the document. | ||
| # Use pipe (|) for multiline entries. | ||
| subtext: The new `init` subcommand initializes a new custom collector | ||
|
|
||
| # Optional: The change log or logs in which this entry should be included. | ||
| # e.g. '[user]' or '[user, api]' | ||
| # Include 'user' if the change is relevant to end users. | ||
| # Include 'api' if there is a change to a library API. | ||
| # Default: '[user]' | ||
| change_logs: [user] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # tmp folder used by tests | ||
| tmp/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package internal // import "go.opentelemetry.io/collector/cmd/builder/internal" | ||
|
|
||
| import ( | ||
| "bytes" | ||
| "embed" | ||
| "errors" | ||
| "fmt" | ||
| "os" | ||
| "os/exec" | ||
| "path" | ||
| "path/filepath" | ||
| "text/template" | ||
|
|
||
| "github.com/spf13/cobra" | ||
|
|
||
| "go.opentelemetry.io/collector/cmd/builder/internal/builder" | ||
| ) | ||
|
|
||
| const defaultDescription = "Custom OpenTelemetry Collector" | ||
|
|
||
| //go:embed init/templates/*.tmpl | ||
| var templatesFS embed.FS | ||
|
|
||
| type metadata struct { | ||
| Name string | ||
| Description string | ||
| StableVersion string | ||
| BetaVersion string | ||
| } | ||
|
|
||
| func initCommand() *cobra.Command { | ||
| var outputPath string | ||
|
|
||
| cmd := &cobra.Command{ | ||
| Use: "init", | ||
| Short: "[EXPERIMENTAL] Initializes a new custom collector repository in the provided folder", | ||
| Long: `ocb init initializes a new repository in the provided folder with a manifest to start building a custom Collector. This command is experimental and very likely to change.`, | ||
| Args: cobra.NoArgs, | ||
| RunE: func(_ *cobra.Command, _ []string) error { | ||
| return run(outputPath) | ||
| }, | ||
| } | ||
|
|
||
| cmd.Flags().StringVar(&outputPath, "path", ".", "Output path where the collector repository will be initialized") | ||
|
|
||
| return cmd | ||
| } | ||
|
|
||
| func run(path string) error { | ||
| if path == "" { | ||
| return errors.New("argument must be a folder") | ||
| } | ||
| path, err := filepath.Abs(path) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to get absolute path for %v: %w", path, err) | ||
| } | ||
| err = os.MkdirAll(path, 0o750) | ||
| if err != nil { | ||
| return fmt.Errorf("failed creating folder %v: %w", path, err) | ||
| } | ||
|
|
||
| meta := metadata{ | ||
| Name: filepath.Base(path), | ||
| Description: defaultDescription, | ||
| StableVersion: builder.DefaultStableOtelColVersion, | ||
| BetaVersion: builder.DefaultBetaOtelColVersion, | ||
| } | ||
|
|
||
| err = writeTemplate(path, "manifest.yaml", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing manifest: %w", err) | ||
| } | ||
|
|
||
| err = writeTemplate(path, ".gitignore", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing gitignore: %w", err) | ||
| } | ||
|
|
||
| err = writeTemplate(path, "README.md", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing README: %w", err) | ||
| } | ||
|
|
||
| err = writeTemplate(path, "go.mod", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing go.mod: %w", err) | ||
| } | ||
|
|
||
| err = writeTemplate(path, "Makefile", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing Makefile: %w", err) | ||
| } | ||
|
|
||
| err = writeTemplate(path, "config.yaml", meta) | ||
| if err != nil { | ||
| return fmt.Errorf("failed writing config.yaml: %w", err) | ||
| } | ||
|
|
||
| err = os.MkdirAll(filepath.Join(path, "build"), 0o750) | ||
| if err != nil { | ||
| return fmt.Errorf("failed creating build folder: %w", err) | ||
| } | ||
|
|
||
| err = runTidy(path) | ||
| if err != nil { | ||
| return fmt.Errorf("failed running go mod tidy: %w", err) | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| func writeTemplate(path, fn string, m metadata) error { | ||
| outputFile := filepath.Join(path, fn) | ||
|
|
||
| content, err := executeTemplate(fn+".tmpl", m) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| return os.WriteFile(outputFile, content, 0o600) | ||
| } | ||
|
|
||
| func executeTemplate(tmplFile string, m metadata) ([]byte, error) { | ||
| tmplPath := path.Join("init/templates", tmplFile) | ||
| tmpl := template.Must(template.New(tmplFile).ParseFS(templatesFS, tmplPath)) | ||
| buf := bytes.Buffer{} | ||
|
|
||
| if err := tmpl.Execute(&buf, m); err != nil { | ||
| return []byte{}, fmt.Errorf("failed executing template: %w", err) | ||
| } | ||
| return buf.Bytes(), nil | ||
| } | ||
|
|
||
| func runTidy(path string) error { | ||
| cmd := exec.Command("go", "mod", "tidy") | ||
| cmd.Dir = path | ||
| output, err := cmd.CombinedOutput() | ||
| if err != nil { | ||
| return fmt.Errorf("%w (%s)", err, string(output)) | ||
| } | ||
| return nil | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package internal // import "go.opentelemetry.io/collector/cmd/builder/internal" | ||
|
|
||
| import ( | ||
| "path/filepath" | ||
| "testing" | ||
|
|
||
| "github.com/knadh/koanf/parsers/yaml" | ||
| "github.com/knadh/koanf/providers/file" | ||
| "github.com/knadh/koanf/v2" | ||
| "github.com/spf13/cobra" | ||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
|
|
||
| "go.opentelemetry.io/collector/cmd/builder/internal/builder" | ||
| ) | ||
|
|
||
| func TestInitCommand(t *testing.T) { | ||
| cmd := initCommand() | ||
|
|
||
| assert.NotNil(t, cmd) | ||
| assert.IsType(t, &cobra.Command{}, cmd) | ||
| assert.Equal(t, "init", cmd.Use) | ||
| } | ||
|
|
||
| func TestRunInit(t *testing.T) { | ||
| for _, tt := range []struct { | ||
| name string | ||
| buildPath func(string) string | ||
|
|
||
| wantErr string | ||
| }{ | ||
| { | ||
| name: "without a path", | ||
| buildPath: func(string) string { return "" }, | ||
| wantErr: "argument must be a folder", | ||
| }, | ||
| { | ||
| name: "with a relative path", | ||
| buildPath: func(string) string { return "./tmp/init" }, | ||
| wantErr: "", | ||
| }, | ||
| { | ||
| name: "with an absolute path", | ||
| buildPath: func(dir string) string { return dir }, | ||
| wantErr: "", | ||
| }, | ||
| } { | ||
| t.Run(tt.name, func(t *testing.T) { | ||
| tmpdir := filepath.Join(t.TempDir(), "init") | ||
| path := tt.buildPath(tmpdir) | ||
| err := run(path) | ||
|
|
||
| if tt.wantErr == "" { | ||
| require.NoError(t, err) | ||
| validateCollector(t, path) | ||
| } else { | ||
| require.ErrorContains(t, err, tt.wantErr) | ||
| } | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| func validateCollector(t *testing.T, path string) { | ||
| require.FileExists(t, filepath.Join(path, ".gitignore")) | ||
| require.FileExists(t, filepath.Join(path, "README.md")) | ||
| require.FileExists(t, filepath.Join(path, "manifest.yaml")) | ||
| require.FileExists(t, filepath.Join(path, "go.mod")) | ||
| require.FileExists(t, filepath.Join(path, "go.sum")) | ||
| require.FileExists(t, filepath.Join(path, "Makefile")) | ||
| require.FileExists(t, filepath.Join(path, "config.yaml")) | ||
|
|
||
| k := koanf.New(".") | ||
| err := k.Load(file.Provider(filepath.Join(path, "manifest.yaml")), yaml.Parser()) | ||
| require.NoError(t, err) | ||
|
|
||
| cfg := builder.Config{} | ||
| err = k.UnmarshalWithConf("", &cfg, koanf.UnmarshalConf{ | ||
| Tag: "mapstructure", | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| assert.Equal(t, "init", cfg.Distribution.Name) | ||
| assert.Equal(t, defaultDescription, cfg.Distribution.Description) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| build/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| GOMODULES := $(shell find . -mindepth 2 \ | ||
| -type f \ | ||
| -name "go.mod" \ | ||
| -not -path "./internal/tools/*" \ | ||
| -exec dirname {} \; | sort ) | ||
|
|
||
|
|
||
| .PHONY: generate | ||
| generate: | ||
| go tool go.opentelemetry.io/collector/cmd/builder --verbose --config manifest.yaml | ||
| cd build/collector && go mod tidy | ||
|
|
||
| .PHONY: run | ||
| run: generate | ||
| @cd build/collector && \ | ||
| ./{{.Name}} --config ../../config.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # {{.Name}} Collector | ||
|
|
||
| Welcome to your new custom OpenTelemetry Collector! | ||
| This Collector allows you to configure the components you wish to use, and only those. | ||
|
|
||
| Components can come from the ones provided by the OpenTelemetry organization in | ||
| the | ||
| [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) | ||
| repository, or from any other source, including your [custom, possibly private | ||
| code](https://opentelemetry.io/docs/collector/extend/custom-component/). The | ||
| only requirement is that they match the Go interface for that component, so the | ||
| Collector can run them. | ||
|
|
||
| ## Important files | ||
|
|
||
| Edit these two files to customize your Collector: | ||
|
|
||
| * `manifest.yaml` - The manifest file configures the OpenTelemetry Collector Builder and tells it what modules it needs to enable within your custom collector. | ||
| See [Configure the OpenTelemetry Collector Builder](https://opentelemetry.io/docs/collector/extend/ocb/#configure-the-opentelemetry-collector-builder) for more information. | ||
| * `config.yaml` - The Collector configuration allows you to configure your components and their pipelines. This is where you tell your Collector where it needs to export data. | ||
| See [Collector Configuration](https://opentelemetry.io/docs/collector/configuration/) for more information. | ||
|
|
||
| ## Running your custom Collector | ||
|
|
||
| You can already run your custom Collector, though it only includes the OTLP | ||
| receiver and exporter. | ||
|
|
||
| Use the following make command: | ||
|
|
||
| ``` | ||
| make run | ||
| ``` | ||
|
|
||
| This will build your collector inside the `build/collector` folder, and then | ||
| run the compiled binary. | ||
|
|
||
| With the `make generate` command, you can also build the collector but not run it. | ||
|
|
||
| ## Next | ||
|
|
||
| To learn more, read the [Extend the | ||
| Collector](https://opentelemetry.io/docs/collector/extend/) documentation, | ||
| which will give you pointers to configure your new custom Collector, and to | ||
| build custom components. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.