diff --git a/CHANGELOG.md b/CHANGELOG.md index d33a7f61221..d0743c24dfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ### Added +- Added the [`generate csv --deploy-dir --apis-dir --crd-dir`](doc/cli/operator-sdk_generate_csv.md#options) flags to allow configuring input locations for operator manifests and API types directories to the CSV generator in lieu of a config. See the CLI reference doc or `generate csv -h` help text for more details. ([#2511](https://github.com/operator-framework/operator-sdk/pull/2511)) +- Added the [`generate csv --output-dir`](doc/cli/operator-sdk_generate_csv.md#options) flag to allow configuring the output location for the catalog directory. ([#2511](https://github.com/operator-framework/operator-sdk/pull/2511)) - The flag `--watch-namespace` and `--operator-namespace` was added to `operator-sdk run --local`, `operator-sdk test --local` and `operator-sdk cleanup` commands in order to replace the flag `--namespace` which was deprecated.([#2617](https://github.com/operator-framework/operator-sdk/pull/2617)) - The methods `ctx.GetOperatorNamespace()` and `ctx.GetWatchNamespace()` was added `pkg/test` in order to replace `ctx.GetNamespace()` which is deprecated. ([#2617](https://github.com/operator-framework/operator-sdk/pull/2617)) - The `--crd-version` flag was added to the `new`, `add api`, `add crd`, and `generate crds` commands so that users can opt-in to `v1` CRDs. ([#2684](https://github.com/operator-framework/operator-sdk/pull/2684)) @@ -29,6 +31,7 @@ - **Breaking Change:** remove `pkg/restmapper` which was deprecated in `v0.14.0`. Projects that use this package must switch to the `DynamicRESTMapper` implementation in [controller-runtime](https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/client/apiutil#NewDynamicRESTMapper). ([#2544](https://github.com/operator-framework/operator-sdk/pull/2544)) - **Breaking Change:** remove deprecated `operator-sdk generate openapi` subcommand. ([#2740](https://github.com/operator-framework/operator-sdk/pull/2740)) +- **Breaking Change:** Removed CSV configuration file support (defaulting to deploy/olm-catalog/csv-config.yaml) in favor of specifying inputs to the generator via [`generate csv --deploy-dir --apis-dir --crd-dir`](doc/cli/operator-sdk_generate_csv.md#options), and configuring output locations via [`generate csv --output-dir`](doc/cli/operator-sdk_generate_csv.md#options). ([#2511](https://github.com/operator-framework/operator-sdk/pull/2511)) ### Bug Fixes diff --git a/cmd/operator-sdk/bundle/create.go b/cmd/operator-sdk/bundle/create.go index d3e6ddf8e7f..517fe7c2566 100644 --- a/cmd/operator-sdk/bundle/create.go +++ b/cmd/operator-sdk/bundle/create.go @@ -22,7 +22,7 @@ import ( "os" "path/filepath" - catalog "github.com/operator-framework/operator-sdk/internal/scaffold/olm-catalog" + catalog "github.com/operator-framework/operator-sdk/internal/generate/olm-catalog" "github.com/operator-framework/operator-sdk/internal/util/projutil" "github.com/operator-framework/operator-registry/pkg/lib/bundle" diff --git a/cmd/operator-sdk/generate/csv.go b/cmd/operator-sdk/generate/csv.go index 70b5c9cb69c..fd970de1bd6 100644 --- a/cmd/operator-sdk/generate/csv.go +++ b/cmd/operator-sdk/generate/csv.go @@ -17,13 +17,12 @@ package generate import ( "fmt" "io/ioutil" + "os" "path/filepath" "github.com/operator-framework/operator-sdk/internal/generate/gen" gencatalog "github.com/operator-framework/operator-sdk/internal/generate/olm-catalog" "github.com/operator-framework/operator-sdk/internal/scaffold" - "github.com/operator-framework/operator-sdk/internal/scaffold/input" - catalog "github.com/operator-framework/operator-sdk/internal/scaffold/olm-catalog" "github.com/operator-framework/operator-sdk/internal/util/fileutil" "github.com/operator-framework/operator-sdk/internal/util/k8sutil" "github.com/operator-framework/operator-sdk/internal/util/projutil" @@ -37,8 +36,11 @@ type csvCmd struct { csvVersion string csvChannel string fromVersion string - csvConfigPath string operatorName string + outputDir string + deployDir string + apisDir string + crdDir string updateCRDs bool defaultChannel bool } @@ -55,18 +57,104 @@ A CSV semantic version is supplied via the --csv-version flag. If your operator has already generated a CSV manifest you want to use as a base, supply its version to --from-version. Otherwise the SDK will scaffold a new CSV manifest. -Configure CSV generation by writing a config file 'deploy/olm-catalog/csv-config.yaml`, - RunE: func(cmd *cobra.Command, args []string) error { - // The CSV generator assumes that the deploy and pkg directories are - // present at runtime, so this command must be run in a project's root. - projutil.MustInProjectRoot() +CSV input flags: + --deploy-dir: + The CSV's install strategy and permissions will be generated from the operator manifests + (Deployment and Role/ClusterRole) present in this directory. + + --apis-dir: + The CSV annotation comments will be parsed from the Go types under this path to + fill out metadata for owned APIs in spec.customresourcedefinitions.owned. + + --crd-dir: + The CSV's spec.customresourcedefinitions.owned field is generated from the CRD manifests + in this path.These CRD manifests are also copied over to the bundle directory if --update-crds is set. + Additionally the CR manifests will be used to populate the CSV example CRs. +`, + Example: ` + ##### Generate CSV from default input paths ##### + $ tree pkg/apis/ deploy/ + pkg/apis/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + deploy/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree deploy/ + deploy/ + ... + ├── olm-catalog + │   └── memcached-operator + │   ├── 0.0.1 + │   │   ├── cache.example.com_memcacheds_crd.yaml + │   │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + │   └── memcached-operator.package.yaml + ... + + + ##### Generate CSV from custom input paths ##### + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds \ + --deploy-dir=config --apis-dir=api --output-dir=production + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree config/ api/ production/ + config/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + api/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + production/ + └── olm-catalog + └── memcached-operator + ├── 0.0.1 + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + └── memcached-operator.package.yaml +`, + + RunE: func(cmd *cobra.Command, args []string) error { if len(args) != 0 { return fmt.Errorf("command %s doesn't accept any arguments", cmd.CommandPath()) } if err := c.validate(); err != nil { return fmt.Errorf("error validating command flags: %v", err) } + + if err := projutil.CheckProjectRoot(); err != nil { + log.Warn("Could not detect project root. Ensure that this command " + + "runs from the project root directory.") + } + + // Default for crd dir if unset + if c.crdDir == "" { + c.crdDir = c.deployDir + } if err := c.run(); err != nil { log.Fatal(err) } @@ -74,14 +162,28 @@ Configure CSV generation by writing a config file 'deploy/olm-catalog/csv-config }, } - cmd.Flags().StringVar(&c.csvVersion, "csv-version", "", "Semantic version of the CSV") + cmd.Flags().StringVar(&c.csvVersion, "csv-version", "", + "Semantic version of the CSV") if err := cmd.MarkFlagRequired("csv-version"); err != nil { log.Fatalf("Failed to mark `csv-version` flag for `generate csv` subcommand as required: %v", err) } cmd.Flags().StringVar(&c.fromVersion, "from-version", "", "Semantic version of an existing CSV to use as a base") - cmd.Flags().StringVar(&c.csvConfigPath, "csv-config", "", - "Path to CSV config file. Defaults to deploy/olm-catalog/csv-config.yaml") + + // TODO: Allow multiple paths + // Deployment and RBAC manifests might be in different dirs e.g kubebuilder + cmd.Flags().StringVar(&c.deployDir, "deploy-dir", "deploy", + `Project relative path to root directory for operator manifests (Deployment and RBAC)`) + cmd.Flags().StringVar(&c.apisDir, "apis-dir", filepath.Join("pkg", "apis"), + `Project relative path to root directory for API type defintions`) + // TODO: Allow multiple paths + // CRD and CR manifests might be in different dirs e.g kubebuilder + cmd.Flags().StringVar(&c.crdDir, "crd-dir", filepath.Join("deploy", "crds"), + `Project relative path to root directory for CRD and CR manifests`) + + cmd.Flags().StringVar(&c.outputDir, "output-dir", scaffold.DeployDir, + "Base directory to output generated CSV. The resulting CSV bundle directory "+ + "will be \"/olm-catalog//\".") cmd.Flags().BoolVar(&c.updateCRDs, "update-crds", false, "Update CRD manifests in deploy/{operator-name}/{csv-version} the using latest API's") cmd.Flags().StringVar(&c.operatorName, "operator-name", "", @@ -96,61 +198,45 @@ Configure CSV generation by writing a config file 'deploy/olm-catalog/csv-config } func (c csvCmd) run() error { - - absProjectPath := projutil.MustGetwd() - cfg := &input.Config{ - AbsProjectPath: absProjectPath, - ProjectName: filepath.Base(absProjectPath), - } - if projutil.IsOperatorGo() { - cfg.Repo = projutil.GetGoPkg() - } - log.Infof("Generating CSV manifest version %s", c.csvVersion) - csvCfg, err := catalog.GetCSVConfig(c.csvConfigPath) - if err != nil { - return err - } if c.operatorName == "" { - // Use config operator name if not set by CLI, i.e. prefer CLI value over - // config value. - if c.operatorName = csvCfg.OperatorName; c.operatorName == "" { - // Default to using project name if both are empty. - c.operatorName = filepath.Base(absProjectPath) - } - } - - s := &scaffold.Scaffold{} - csv := &catalog.CSV{ - CSVVersion: c.csvVersion, - FromVersion: c.fromVersion, - ConfigFilePath: c.csvConfigPath, - OperatorName: c.operatorName, + c.operatorName = filepath.Base(projutil.MustGetwd()) } - err = s.Execute(cfg, csv) - if err != nil { - return fmt.Errorf("catalog scaffold failed: %v", err) + cfg := gen.Config{ + OperatorName: c.operatorName, + // TODO(hasbro17): Remove the Input key map when the Generator input keys + // are removed in favour of config fields in the csvGenerator + Inputs: map[string]string{ + gencatalog.DeployDirKey: c.deployDir, + gencatalog.APIsDirKey: c.apisDir, + gencatalog.CRDsDirKey: c.crdDir, + }, + OutputDir: c.outputDir, } - gcfg := gen.Config{ - OperatorName: c.operatorName, - OutputDir: filepath.Join(gencatalog.OLMCatalogDir, c.operatorName), + csv := gencatalog.NewCSV(cfg, c.csvVersion, c.fromVersion) + if err := csv.Generate(); err != nil { + return fmt.Errorf("error generating CSV: %v", err) } - pkg := gencatalog.NewPackageManifest(gcfg, c.csvVersion, c.csvChannel, c.defaultChannel) + pkg := gencatalog.NewPackageManifest(cfg, c.csvVersion, c.csvChannel, c.defaultChannel) if err := pkg.Generate(); err != nil { return fmt.Errorf("error generating package manifest: %v", err) } // Write CRD's to the new or updated CSV package dir. if c.updateCRDs { - input, err := csv.GetInput() + crdManifestSet, err := findCRDFileSet(c.crdDir) if err != nil { - return err + return fmt.Errorf("failed to update CRD's: %v", err) } - err = writeCRDsToDir(csvCfg.CRDCRPaths, filepath.Dir(input.Path)) - if err != nil { - return err + // TODO: This path should come from the CSV generator field csvOutputDir + bundleDir := filepath.Join(c.outputDir, gencatalog.OLMCatalogChildDir, c.operatorName, c.csvVersion) + for path, b := range crdManifestSet { + path = filepath.Join(bundleDir, path) + if err = ioutil.WriteFile(path, b, fileutil.DefaultFileMode); err != nil { + return fmt.Errorf("failed to update CRD's: %v", err) + } } } @@ -191,25 +277,45 @@ func validateVersion(version string) error { return nil } -func writeCRDsToDir(crdPaths []string, toDir string) error { - for _, p := range crdPaths { - b, err := ioutil.ReadFile(p) +// findCRDFileSet searches files in the given directory path for CRD manifests, +// returning a map of paths to file contents. +func findCRDFileSet(crdDir string) (map[string][]byte, error) { + crdFileSet := map[string][]byte{} + info, err := os.Stat(crdDir) + if err != nil { + return nil, err + } + if !info.IsDir() { + return nil, fmt.Errorf("crd's must be read from a directory. %s is a file", crdDir) + } + files, err := ioutil.ReadDir(crdDir) + if err != nil { + return nil, err + } + + wd := projutil.MustGetwd() + for _, f := range files { + if f.IsDir() { + continue + } + + crdPath := filepath.Join(wd, crdDir, f.Name()) + b, err := ioutil.ReadFile(crdPath) if err != nil { - return err + return nil, fmt.Errorf("error reading manifest %s: %v", crdPath, err) } + // Skip files in crdsDir that aren't k8s manifests since we do not know + // what other files are in crdsDir. typeMeta, err := k8sutil.GetTypeMetaFromBytes(b) if err != nil { - return fmt.Errorf("error in %s : %v", p, err) + log.Debugf("Skipping non-manifest file %s: %v", crdPath, err) + continue } if typeMeta.Kind != "CustomResourceDefinition" { + log.Debugf("Skipping non CRD manifest %s", crdPath) continue } - - path := filepath.Join(toDir, filepath.Base(p)) - err = ioutil.WriteFile(path, b, fileutil.DefaultFileMode) - if err != nil { - return err - } + crdFileSet[filepath.Base(crdPath)] = b } - return nil + return crdFileSet, nil } diff --git a/cmd/operator-sdk/run/cmd.go b/cmd/operator-sdk/run/cmd.go index 0691737a6e8..86069455618 100644 --- a/cmd/operator-sdk/run/cmd.go +++ b/cmd/operator-sdk/run/cmd.go @@ -19,8 +19,8 @@ import ( "fmt" "path/filepath" + olmcatalog "github.com/operator-framework/operator-sdk/internal/generate/olm-catalog" olmoperator "github.com/operator-framework/operator-sdk/internal/olm/operator" - olmcatalog "github.com/operator-framework/operator-sdk/internal/scaffold/olm-catalog" k8sinternal "github.com/operator-framework/operator-sdk/internal/util/k8sutil" "github.com/operator-framework/operator-sdk/internal/util/projutil" aoflags "github.com/operator-framework/operator-sdk/pkg/ansible/flags" diff --git a/doc/cli/operator-sdk_generate_csv.md b/doc/cli/operator-sdk_generate_csv.md index 1b20934ca18..57d80b6d193 100644 --- a/doc/cli/operator-sdk_generate_csv.md +++ b/doc/cli/operator-sdk_generate_csv.md @@ -11,22 +11,109 @@ A CSV semantic version is supplied via the --csv-version flag. If your operator has already generated a CSV manifest you want to use as a base, supply its version to --from-version. Otherwise the SDK will scaffold a new CSV manifest. -Configure CSV generation by writing a config file 'deploy/olm-catalog/csv-config.yaml +CSV input flags: + --deploy-dir: + The CSV's install strategy and permissions will be generated from the operator manifests + (Deployment and Role/ClusterRole) present in this directory. + + --apis-dir: + The CSV annotation comments will be parsed from the Go types under this path to + fill out metadata for owned APIs in spec.customresourcedefinitions.owned. + + --crd-dir: + The CSV's spec.customresourcedefinitions.owned field is generated from the CRD manifests + in this path.These CRD manifests are also copied over to the bundle directory if --update-crds is set. + Additionally the CR manifests will be used to populate the CSV example CRs. + ``` operator-sdk generate csv [flags] ``` +### Examples + +``` + + ##### Generate CSV from default input paths ##### + $ tree pkg/apis/ deploy/ + pkg/apis/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + deploy/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree deploy/ + deploy/ + ... + ├── olm-catalog + │   └── memcached-operator + │   ├── 0.0.1 + │   │   ├── cache.example.com_memcacheds_crd.yaml + │   │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + │   └── memcached-operator.package.yaml + ... + + + + ##### Generate CSV from custom input paths ##### + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds \ + --deploy-dir=config --apis-dir=api --output-dir=production + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree config/ api/ production/ + config/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + api/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + production/ + └── olm-catalog + └── memcached-operator + ├── 0.0.1 + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + └── memcached-operator.package.yaml + +``` + ### Options ``` + --apis-dir string Project relative path to root directory for API type defintions (default "pkg/apis") + --crd-dir string Project relative path to root directory for CRD and CR manifests (default "deploy/crds") --csv-channel string Channel the CSV should be registered under in the package manifest - --csv-config string Path to CSV config file. Defaults to deploy/olm-catalog/csv-config.yaml --csv-version string Semantic version of the CSV --default-channel Use the channel passed to --csv-channel as the package manifests' default channel. Only valid when --csv-channel is set + --deploy-dir string Project relative path to root directory for operator manifests (Deployment and RBAC) (default "deploy") --from-version string Semantic version of an existing CSV to use as a base -h, --help help for csv --operator-name string Operator name to use while generating CSV + --output-dir string Base directory to output generated CSV. The resulting CSV bundle directory will be "/olm-catalog//". (default "deploy") --update-crds Update CRD manifests in deploy/{operator-name}/{csv-version} the using latest API's ``` diff --git a/doc/user/olm-catalog/generating-a-csv.md b/doc/user/olm-catalog/generating-a-csv.md index 4db5492e0d8..3f6782a25a1 100644 --- a/doc/user/olm-catalog/generating-a-csv.md +++ b/doc/user/olm-catalog/generating-a-csv.md @@ -10,35 +10,23 @@ This document describes how to manage the following lifecycle for your Operator ## Configuration -Operator SDK projects have an expected [project layout][doc-project-layout]. In particular, a few manifests are expected to be present in the `deploy` directory: +### Inputs -* Roles: `role.yaml` -* Deployments: `operator.yaml` -* Custom Resources (CR's): `crds/___cr.yaml` -* Custom Resource Definitions (CRD's): `crds/__crd.yaml`. +The CSV generator requires certain inputs to construct a CSV manifest. -`generate csv` reads these files and adds their data to a CSV in an alternate form. +1. Path to the operator manifests root directory. By default `generate csv` extracts manifests from files in `deploy/` for the following kinds and adds them to the CSV. Use the `--deploy-dir` flag to change this path. + * Roles: `role.yaml` + * ClusterRoles: `cluster_role.yaml` + * Deployments: `operator.yaml` + * Custom Resources (CR's): `crds/___cr.yaml` + * CustomResourceDefinitions (CRD's): `crds/__crd.yaml` +2. Path to API types root directory. The CSV generator also parses the [CSV annotations][csv-annotations] from the API type definitions to populate certain CSV fields. By default the API types directory is `pkg/apis/`. Use the `--apis-dir` flag to change this path. The CSV generator expects either of the following layouyts for the API types directory + * Mulitple groups: `///` + * Single groups: `//` -The following example config containing default values should be copied and written to `deploy/olm-catalog/csv-config.yaml`: +### Output -```yaml -crd-cr-paths: -- deploy/crds -operator-path: deploy/operator.yaml -role-paths: -- deploy/role.yaml -``` - -Explanation of all config fields: - -- `crd-cr-paths`: list of strings - a list of CRD and CR manifest file/directory paths. Defaults to `[deploy/crds]`. -- `operator-path`: string - the operator `Deployment` manifest file path. Defaults to `deploy/operator.yaml`. -- `role-paths`: list of strings - Role and ClusterRole manifest file paths. Defaults to `[deploy/role.yaml]`. -- `operator-name`: string - the name used to create the CSV and manifest file names. Defaults to the project's name. - -**Note**: The [design doc][doc-csv-design] has outdated field information which should not be referenced. - -Fields in this config file can be modified to point towards alternate manifest locations, and passed to `generate csv --csv-config=` to configure CSV generation. For example, if I have one set of production CR/CRD manifests under `deploy/crds/production`, and a set of test manifests under `deploy/crds/test`, and I only want to include production manifests in my CSV, I can set `crd-cr-paths: [deploy/crds/production]`. `generate csv` will then ignore `deploy/crds/test` when getting CR/CRD data. +By default `generate csv` will generate the catalog bundle directory `olm-catalog/...` under `deploy/`. To change where the CSV bundle directory is generated use the `--ouput-dir` flag. ## Versioning @@ -46,7 +34,7 @@ CSV's are versioned in path, file name, and in their `metadata.name` field. For `generate csv` allows you to upgrade your CSV using the `--from-version` flag. If you have an existing CSV with version `0.0.1` and want to write a new version `0.0.2`, you can run `operator-sdk generate csv --csv-version 0.0.2 --from-version 0.0.1`. This will write a new CSV manifest to `deploy/olm-catalog//0.0.2/.v0.0.2.clusterserviceversion.yaml` containing user-defined data from `0.0.1` and any modifications you've made to `roles.yaml`, `operator.yaml`, CR's, or CRD's. -The SDK can manage CRD's in your Operator bundle as well. You can pass the `--update-crds` flag to `generate csv` to add or update your CRD's in your bundle by copying manifests pointed to by `crd-cr-paths` in your config. CRD's in a bundle are not updated by default. +The SDK can manage CRD's in your Operator bundle as well. You can pass the `--update-crds` flag to `generate csv` to add or update your CRD's in your bundle by copying manifests in `deploy/crds` to your bundle. CRD's in a bundle are not updated by default. ## First Generation @@ -86,7 +74,7 @@ Be sure to include the `--update-crds` flag if you want to add CRD's to your bun Below are two lists of fields: the first is a list of all fields the SDK and OLM expect in a CSV, and the second are optional. -Several fields require user input (labeled _user_) or a [code annotation][code-annotations] (labeled _annotation_). This list may change as the SDK becomes better at generating CSV's. +Several fields require user input (labeled _user_) or a [CSV annotation][csv-annotations] (labeled _annotation_). This list may change as the SDK becomes better at generating CSV's. Required: @@ -130,10 +118,9 @@ Optional: [doc-csv]:https://github.com/operator-framework/operator-lifecycle-manager/blob/4197455/Documentation/design/building-your-csv.md [olm]:https://github.com/operator-framework/operator-lifecycle-manager [generate-csv-cli]:../../cli/operator-sdk_generate_csv.md -[doc-project-layout]:../../project_layout.md [doc-csv-design]:../../design/milestone-0.2.0/csv-generation.md [doc-bundle]:https://github.com/operator-framework/operator-registry/blob/6893d19/README.md#manifest-format [x-desc-list]:https://github.com/openshift/console/blob/70bccfe/frontend/public/components/operator-lifecycle-manager/descriptors/types.ts#L3-L35 [install-modes]:https://github.com/operator-framework/operator-lifecycle-manager/blob/4197455/Documentation/design/building-your-csv.md#operator-metadata [olm-capabilities]:../../images/operator-capability-level.png -[code-annotations]:../../proposals/sdk-code-annotations.md +[csv-annotations]: ./csv-annotations.md diff --git a/internal/generate/gen/config.go b/internal/generate/gen/config.go index 5ff046368df..d249f80a4a3 100644 --- a/internal/generate/gen/config.go +++ b/internal/generate/gen/config.go @@ -14,13 +14,8 @@ package gen -import ( - "path/filepath" - "strings" - - "github.com/operator-framework/operator-sdk/internal/util/projutil" -) - +// TODO(hasbro17/estroz): Remove the generator config in favor of generator +// specific option structs configured with Inputs and OutputDir. // Config configures a generator with common operator project information. type Config struct { // OperatorName is the operator's name, ex. app-operator @@ -31,46 +26,7 @@ type Config struct { // on-disk inputs are required. If not set, a default is used on a // per-generator basis. Inputs map[string]string - // OutputDir is a dir in which to generate output files. If not set, a - // default is used on a per-generator basis. + // OutputDir is the root directory where the output files will be generated. + // If not set, a default is used on a per-generator basis. OutputDir string - // Filters is a set of functional filters for paths that a generator may - // encounter while gathering data for generation. Filters provides - // fine-grained control over Inputs, since often those paths are often - // top-level directories. - Filters FilterFuncs -} - -// FilterFuncs is a slice of filter funcs. -type FilterFuncs []func(string) bool - -// MakeFilters creates a set of closures around each path in paths. -// If the argument to a closure has a prefix of path, it returns true. -func MakeFilters(paths ...string) (filters FilterFuncs) { - pathSet := map[string]struct{}{} - for _, path := range paths { - pathSet[filepath.Clean(path)] = struct{}{} - } - wd := projutil.MustGetwd() + string(filepath.Separator) - for path := range pathSet { - // Copy the string for the closure. - pb := strings.Builder{} - pb.WriteString(path) - filters = append(filters, func(p string) bool { - // Handle absolute paths referencing the project directory. - p = strings.TrimPrefix(p, wd) - return strings.HasPrefix(filepath.Clean(p), pb.String()) - }) - } - return filters -} - -// SatisfiesAny returns true if path passes any filter in funcs. -func (funcs FilterFuncs) SatisfiesAny(path string) bool { - for _, f := range funcs { - if f(path) { - return true - } - } - return false } diff --git a/internal/generate/gen/config_test.go b/internal/generate/gen/config_test.go deleted file mode 100644 index 2ad25969880..00000000000 --- a/internal/generate/gen/config_test.go +++ /dev/null @@ -1,63 +0,0 @@ -// Copyright 2020 The Operator-SDK Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package gen - -import ( - "testing" - - "github.com/stretchr/testify/assert" -) - -func TestFilterFuncs(t *testing.T) { - cases := []struct { - name string - paths []string - wantedSatPaths map[string]bool - }{ - { - "empty filters with one path", - nil, - map[string]bool{"notexist": false}, - }, - { - "two filters with no matching paths", - []string{"key1", "key2"}, - map[string]bool{"notexist": false}, - }, - { - "multiple pathed filters with multiple matching paths", - []string{"key1/key2", "key3", "/abs/path/to/something"}, - map[string]bool{ - "key3": true, - "/abs/path/to/something/else": true, - "key2": false, - "path/to": false, - }, - }, - } - for _, c := range cases { - filters := MakeFilters(c.paths...) - for path, wantedSat := range c.wantedSatPaths { - t.Run(c.name+": "+path, func(t *testing.T) { - isSat := filters.SatisfiesAny(path) - if wantedSat { - assert.True(t, isSat) - } else { - assert.False(t, isSat) - } - }) - } - } -} diff --git a/internal/generate/olm-catalog/csv.go b/internal/generate/olm-catalog/csv.go index 5a70c3fd951..9d0052d5958 100644 --- a/internal/generate/olm-catalog/csv.go +++ b/internal/generate/olm-catalog/csv.go @@ -15,15 +15,488 @@ package olmcatalog import ( + "fmt" + "io/ioutil" + "os" "path/filepath" + "regexp" + "sort" + "strings" + "github.com/operator-framework/operator-sdk/internal/generate/gen" "github.com/operator-framework/operator-sdk/internal/scaffold" + "github.com/operator-framework/operator-sdk/internal/util/fileutil" + "github.com/operator-framework/operator-sdk/internal/util/k8sutil" + "github.com/operator-framework/operator-sdk/internal/util/projutil" + "github.com/operator-framework/operator-sdk/internal/util/yamlutil" + + "github.com/blang/semver" + "github.com/ghodss/yaml" + olmapiv1alpha1 "github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1" + olmversion "github.com/operator-framework/operator-lifecycle-manager/pkg/lib/version" + "github.com/pkg/errors" + log "github.com/sirupsen/logrus" + "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" ) const ( - OLMCatalogDir = scaffold.DeployDir + string(filepath.Separator) + "olm-catalog" + OLMCatalogChildDir = "olm-catalog" + // OLMCatalogDir is the default location for OLM catalog directory. + OLMCatalogDir = scaffold.DeployDir + string(filepath.Separator) + OLMCatalogChildDir + csvYamlFileExt = ".clusterserviceversion.yaml" + + // Input keys for CSV generator whose values are the filepaths for the respective input directories + + // DeployDirKey is for the location of the operator manifests directory e.g "deploy/production" + // The Deployment and RBAC manifests from this directory will be used to populate the CSV + // install strategy: spec.install + DeployDirKey = "deploy" + // APIsDirKey is for the location of the API types directory e.g "pkg/apis" + // The CSV annotation comments will be parsed from the types under this path. + APIsDirKey = "apis" + // CRDsDirKey is for the location of the CRD manifests directory e.g "deploy/crds" + // Both the CRD and CR manifests from this path will be used to populate CSV fields + // metadata.annotations.alm-examples for CR examples + // and spec.customresourcedefinitions.owned for owned CRDs + CRDsDirKey = "crds" ) +type csvGenerator struct { + gen.Config + // csvVersion is the CSV current version. + csvVersion string + // fromVersion is the CSV version from which to build a new CSV. A CSV + // manifest with this version should exist at: + // deploy/olm-catalog/{from_version}/operator-name.v{from_version}.{csvYamlFileExt} + fromVersion string + // existingCSVBundleDir is set if the generator needs to update from + // an existing CSV bundle directory + existingCSVBundleDir string + // csvOutputDir is the bundle directory filepath where the CSV will be generated + // This is set according to the generator's OutputDir + csvOutputDir string +} + +func NewCSV(cfg gen.Config, csvVersion, fromVersion string) gen.Generator { + g := csvGenerator{ + Config: cfg, + csvVersion: csvVersion, + fromVersion: fromVersion, + } + if g.Inputs == nil { + g.Inputs = map[string]string{} + } + + // The olm-catalog directory location depends on where the output directory is set. + if g.OutputDir == "" { + g.OutputDir = scaffold.DeployDir + } + // Set the CSV bundle dir output path under the generator's OutputDir + olmCatalogDir := filepath.Join(g.OutputDir, OLMCatalogChildDir) + g.csvOutputDir = filepath.Join(olmCatalogDir, g.OperatorName, g.csvVersion) + + bundleParentDir := filepath.Join(olmCatalogDir, g.OperatorName) + if isBundleDirExist(bundleParentDir, g.fromVersion) { + // Upgrading a new CSV from previous CSV version + g.existingCSVBundleDir = filepath.Join(bundleParentDir, g.fromVersion) + } else if isBundleDirExist(bundleParentDir, g.csvVersion) { + // Updating an existing CSV version + g.existingCSVBundleDir = filepath.Join(bundleParentDir, g.csvVersion) + } + + if deployDir, ok := g.Inputs[DeployDirKey]; !ok || deployDir == "" { + g.Inputs[DeployDirKey] = scaffold.DeployDir + } + + if apisDir, ok := g.Inputs[APIsDirKey]; !ok || apisDir == "" { + g.Inputs[APIsDirKey] = scaffold.ApisDir + } + + if crdsDir, ok := g.Inputs[CRDsDirKey]; !ok || crdsDir == "" { + g.Inputs[CRDsDirKey] = filepath.Join(g.Inputs[DeployDirKey], "crds") + } + + return g +} + +func isBundleDirExist(parentDir, version string) bool { + // Ensure full path is constructed. + if parentDir == "" || version == "" { + return false + } + bundleDir := filepath.Join(parentDir, version) + _, err := os.Stat(bundleDir) + if err != nil { + if os.IsNotExist(err) { + return false + } + // TODO: return and handle this error + log.Fatalf("Failed to stat existing bundle directory %s: %v", bundleDir, err) + } + return true +} + func getCSVName(name, version string) string { return name + ".v" + version } + +func getCSVFileName(name, version string) string { + return getCSVName(strings.ToLower(name), version) + csvYamlFileExt +} + +// Generate allows a CSV to be written by marshalling +// olmapiv1alpha1.ClusterServiceVersion instead of writing to a template. +func (g csvGenerator) Generate() error { + fileMap, err := g.generate() + if err != nil { + return err + } + if len(fileMap) == 0 { + return errors.New("error generating CSV manifest: no generated file found") + } + + if err = os.MkdirAll(g.csvOutputDir, fileutil.DefaultDirFileMode); err != nil { + return errors.Wrapf(err, "error mkdir %s", g.csvOutputDir) + } + for fileName, b := range fileMap { + path := filepath.Join(g.csvOutputDir, fileName) + log.Debugf("CSV generator writing %s", path) + if err = ioutil.WriteFile(path, b, fileutil.DefaultFileMode); err != nil { + return err + } + } + return nil +} + +func (g csvGenerator) generate() (fileMap map[string][]byte, err error) { + // Get current CSV to update, otherwise start with a fresh CSV. + var csv *olmapiv1alpha1.ClusterServiceVersion + if g.existingCSVBundleDir != "" { + // TODO: If bundle dir exists, but the CSV file does not + // then we should create a new one and not return an error. + if csv, err = getCSVFromDir(g.existingCSVBundleDir); err != nil { + return nil, err + } + // TODO: validate existing CSV. + if err = g.updateCSVVersions(csv); err != nil { + return nil, err + } + } else { + if csv, err = newCSV(g.OperatorName, g.csvVersion); err != nil { + return nil, err + } + } + + if err = g.updateCSVFromManifests(csv); err != nil { + return nil, err + } + + path := getCSVFileName(g.OperatorName, g.csvVersion) + if fields := getEmptyRequiredCSVFields(csv); len(fields) != 0 { + if g.existingCSVBundleDir != "" { + // An existing csv should have several required fields populated. + log.Warnf("Required csv fields not filled in file %s:%s\n", path, joinFields(fields)) + } else { + // A new csv won't have several required fields populated. + // Report required fields to user informationally. + log.Infof("Fill in the following required fields in file %s:%s\n", path, joinFields(fields)) + } + } + + b, err := k8sutil.GetObjectBytes(csv, yaml.Marshal) + if err != nil { + return nil, err + } + fileMap = map[string][]byte{ + path: b, + } + return fileMap, nil +} + +func getCSVFromDir(dir string) (*olmapiv1alpha1.ClusterServiceVersion, error) { + infos, err := ioutil.ReadDir(dir) + if err != nil { + return nil, err + } + for _, info := range infos { + path := filepath.Join(dir, info.Name()) + info, err := os.Stat(path) + if err != nil || info.IsDir() { + // Skip any directories or files accessed in error. + continue + } + b, err := ioutil.ReadFile(path) + if err != nil { + return nil, err + } + typeMeta, err := k8sutil.GetTypeMetaFromBytes(b) + if err != nil { + return nil, err + } + if typeMeta.Kind != olmapiv1alpha1.ClusterServiceVersionKind { + continue + } + csv := &olmapiv1alpha1.ClusterServiceVersion{} + if err := yaml.Unmarshal(b, csv); err != nil { + return nil, errors.Wrapf(err, "error unmarshalling CSV %s", path) + } + return csv, nil + } + return nil, fmt.Errorf("no CSV manifest in %s", dir) +} + +// newCSV sets all csv fields that should be populated by a user +// to sane defaults. +func newCSV(name, version string) (*olmapiv1alpha1.ClusterServiceVersion, error) { + ver, err := semver.Parse(version) + if err != nil { + return nil, err + } + return &olmapiv1alpha1.ClusterServiceVersion{ + TypeMeta: metav1.TypeMeta{ + APIVersion: olmapiv1alpha1.ClusterServiceVersionAPIVersion, + Kind: olmapiv1alpha1.ClusterServiceVersionKind, + }, + ObjectMeta: metav1.ObjectMeta{ + Name: getCSVName(name, version), + Namespace: "placeholder", + Annotations: map[string]string{ + "capabilities": "Basic Install", + }, + }, + Spec: olmapiv1alpha1.ClusterServiceVersionSpec{ + DisplayName: k8sutil.GetDisplayName(name), + Description: "", + Provider: olmapiv1alpha1.AppLink{}, + Maintainers: make([]olmapiv1alpha1.Maintainer, 1), + Links: []olmapiv1alpha1.AppLink{}, + Maturity: "alpha", + Version: olmversion.OperatorVersion{Version: ver}, + Icon: make([]olmapiv1alpha1.Icon, 1), + Keywords: []string{""}, + InstallModes: []olmapiv1alpha1.InstallMode{ + {Type: olmapiv1alpha1.InstallModeTypeOwnNamespace, Supported: true}, + {Type: olmapiv1alpha1.InstallModeTypeSingleNamespace, Supported: true}, + {Type: olmapiv1alpha1.InstallModeTypeMultiNamespace, Supported: false}, + {Type: olmapiv1alpha1.InstallModeTypeAllNamespaces, Supported: true}, + }, + InstallStrategy: olmapiv1alpha1.NamedInstallStrategy{ + StrategyName: olmapiv1alpha1.InstallStrategyNameDeployment, + StrategySpec: olmapiv1alpha1.StrategyDetailsDeployment{}, + }, + }, + }, nil +} + +// TODO: replace with validation library. +func getEmptyRequiredCSVFields(csv *olmapiv1alpha1.ClusterServiceVersion) (fields []string) { + // Metadata + if csv.TypeMeta.APIVersion != olmapiv1alpha1.ClusterServiceVersionAPIVersion { + fields = append(fields, "apiVersion") + } + if csv.TypeMeta.Kind != olmapiv1alpha1.ClusterServiceVersionKind { + fields = append(fields, "kind") + } + if csv.ObjectMeta.Name == "" { + fields = append(fields, "metadata.name") + } + // Spec fields + if csv.Spec.Version.String() == "" { + fields = append(fields, "spec.version") + } + if csv.Spec.DisplayName == "" { + fields = append(fields, "spec.displayName") + } + if csv.Spec.Description == "" { + fields = append(fields, "spec.description") + } + if len(csv.Spec.Keywords) == 0 || len(csv.Spec.Keywords[0]) == 0 { + fields = append(fields, "spec.keywords") + } + if len(csv.Spec.Maintainers) == 0 { + fields = append(fields, "spec.maintainers") + } + if csv.Spec.Provider == (olmapiv1alpha1.AppLink{}) { + fields = append(fields, "spec.provider") + } + if csv.Spec.Maturity == "" { + fields = append(fields, "spec.maturity") + } + + return fields +} + +func joinFields(fields []string) string { + sb := &strings.Builder{} + for _, f := range fields { + sb.WriteString("\n\t" + f) + } + return sb.String() +} + +// updateCSVVersions updates csv's version and data involving the version, +// ex. ObjectMeta.Name, and place the old version in the `replaces` object, +// if there is an old version to replace. +func (g csvGenerator) updateCSVVersions(csv *olmapiv1alpha1.ClusterServiceVersion) error { + + // Old csv version to replace, and updated csv version. + oldVer, newVer := csv.Spec.Version.String(), g.csvVersion + if oldVer == newVer { + return nil + } + + // Replace all references to the old operator name. + oldCSVName := getCSVName(g.OperatorName, oldVer) + oldRe, err := regexp.Compile(fmt.Sprintf("\\b%s\\b", regexp.QuoteMeta(oldCSVName))) + if err != nil { + return errors.Wrapf(err, "error compiling CSV name regexp %s", oldRe.String()) + } + b, err := yaml.Marshal(csv) + if err != nil { + return err + } + newCSVName := getCSVName(g.OperatorName, newVer) + b = oldRe.ReplaceAll(b, []byte(newCSVName)) + *csv = olmapiv1alpha1.ClusterServiceVersion{} + if err = yaml.Unmarshal(b, csv); err != nil { + return errors.Wrapf(err, "error unmarshalling CSV %s after replacing old CSV name", csv.GetName()) + } + + ver, err := semver.Parse(g.csvVersion) + if err != nil { + return err + } + csv.Spec.Version = olmversion.OperatorVersion{Version: ver} + csv.Spec.Replaces = oldCSVName + return nil +} + +// updateCSVFromManifests gathers relevant data from generated and +// user-defined manifests and updates csv. +func (g csvGenerator) updateCSVFromManifests(csv *olmapiv1alpha1.ClusterServiceVersion) (err error) { + kindManifestMap := map[schema.GroupVersionKind][][]byte{} + + // Read CRD and CR manifests from CRD dir + if err := updateFromManifests(g.Inputs[CRDsDirKey], kindManifestMap); err != nil { + return err + } + + // Get owned CRDs from CRD manifests + ownedCRDs, err := getOwnedCRDs(kindManifestMap) + if err != nil { + return err + } + + // Read Deployment and RBAC manifests from Deploy dir + if err := updateFromManifests(g.Inputs[DeployDirKey], kindManifestMap); err != nil { + return err + } + + // Update CSV from all manifest types + crUpdaters := crs{} + for gvk, manifests := range kindManifestMap { + // We don't necessarily care about sorting by a field value, more about + // consistent ordering. + sort.Slice(manifests, func(i int, j int) bool { + return string(manifests[i]) < string(manifests[j]) + }) + switch gvk.Kind { + case "Role": + err = roles(manifests).apply(csv) + case "ClusterRole": + err = clusterRoles(manifests).apply(csv) + case "Deployment": + err = deployments(manifests).apply(csv) + case "CustomResourceDefinition": + err = crds(manifests).apply(csv) + default: + // Only update CR examples for owned CRD types + if _, ok := ownedCRDs[gvk]; ok { + crUpdaters = append(crUpdaters, crs(manifests)...) + } else { + log.Infof("Skipping manifest %s", gvk) + } + } + if err != nil { + return err + } + } + err = updateDescriptions(csv, g.Inputs[APIsDirKey]) + if err != nil { + return fmt.Errorf("error updating CSV customresourcedefinitions: %w", err) + } + // Re-sort CR's since they are appended in random order. + if len(crUpdaters) != 0 { + sort.Slice(crUpdaters, func(i int, j int) bool { + return string(crUpdaters[i]) < string(crUpdaters[j]) + }) + if err = crUpdaters.apply(csv); err != nil { + return err + } + } + return nil +} + +func updateFromManifests(dir string, kindManifestMap map[schema.GroupVersionKind][][]byte) error { + files, err := ioutil.ReadDir(dir) + if err != nil { + return err + } + // Read and scan all files into kindManifestMap + wd := projutil.MustGetwd() + for _, f := range files { + if f.IsDir() { + continue + } + path := filepath.Join(wd, dir, f.Name()) + b, err := ioutil.ReadFile(path) + if err != nil { + return err + } + scanner := yamlutil.NewYAMLScanner(b) + for scanner.Scan() { + manifest := scanner.Bytes() + typeMeta, err := k8sutil.GetTypeMetaFromBytes(manifest) + if err != nil { + log.Infof("No TypeMeta in %s, skipping file", path) + continue + } + + gvk := typeMeta.GroupVersionKind() + kindManifestMap[gvk] = append(kindManifestMap[gvk], manifest) + } + if scanner.Err() != nil { + return scanner.Err() + } + } + return nil +} + +func getOwnedCRDs(kindManifestMap map[schema.GroupVersionKind][][]byte) (map[schema.GroupVersionKind]struct{}, error) { + ownedCRDs := map[schema.GroupVersionKind]struct{}{} + for gvk, manifests := range kindManifestMap { + if gvk.Kind != "CustomResourceDefinition" { + continue + } + // Collect CRD kinds to filter them out from unsupported manifest types. + // The CRD version type doesn't matter as long as it has a group, kind, + // and versions in the expected fields. + for _, manifest := range manifests { + crd := v1beta1.CustomResourceDefinition{} + if err := yaml.Unmarshal(manifest, &crd); err != nil { + return ownedCRDs, err + } + for _, ver := range crd.Spec.Versions { + crGVK := schema.GroupVersionKind{ + Group: crd.Spec.Group, + Version: ver.Name, + Kind: crd.Spec.Names.Kind, + } + ownedCRDs[crGVK] = struct{}{} + } + } + } + return ownedCRDs, nil +} diff --git a/internal/generate/olm-catalog/csv_go_test.go b/internal/generate/olm-catalog/csv_go_test.go new file mode 100644 index 00000000000..c139f957984 --- /dev/null +++ b/internal/generate/olm-catalog/csv_go_test.go @@ -0,0 +1,424 @@ +// Copyright 2020 The Operator-SDK Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package olmcatalog + +import ( + "io/ioutil" + "log" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + gen "github.com/operator-framework/operator-sdk/internal/generate/gen" + "github.com/operator-framework/operator-sdk/internal/scaffold" + "github.com/operator-framework/operator-sdk/internal/util/fileutil" + internalk8sutil "github.com/operator-framework/operator-sdk/internal/util/k8sutil" + "github.com/operator-framework/operator-sdk/internal/util/projutil" + "github.com/operator-framework/operator-sdk/pkg/k8sutil" + + "github.com/blang/semver" + "github.com/ghodss/yaml" + "github.com/stretchr/testify/assert" + appsv1 "k8s.io/api/apps/v1" +) + +const ( + testProjectName = "memcached-operator" + csvVersion = "0.0.3" + fromVersion = "0.0.2" + notExistVersion = "1.0.0" +) + +var ( + testGoDataDir = filepath.Join("..", "testdata", "go") + testNonStandardLayoutDataDir = filepath.Join("..", "testdata", "non-standard-layout") +) + +func chDirWithCleanup(t *testing.T, dataDir string) func() { + wd, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + if err := os.Chdir(dataDir); err != nil { + t.Fatal(err) + } + chDirCleanupFunc := func() { + if err := os.Chdir(wd); err != nil { + t.Fatal(err) + } + } + return chDirCleanupFunc +} + +// TODO: Change to table driven subtests to test out different Inputs/Output for the generator +func TestGenerateNewCSVWithInputsToOutput(t *testing.T) { + // Change directory to project root so the test cases can form the correct pkg imports + cleanupFunc := chDirWithCleanup(t, testNonStandardLayoutDataDir) + defer cleanupFunc() + + // Temporary output dir for generating catalog bundle + outputDir, err := ioutil.TempDir("", t.Name()+"-output-catalog") + if err != nil { + log.Fatal(err) + } + // Clean up output catalog dir + defer func() { + if err := os.RemoveAll(outputDir); err != nil && !os.IsNotExist(err) { + // Not a test failure since files in /tmp will eventually get deleted + t.Logf("Failed to remove tmp generated catalog directory (%s): %v", outputDir, err) + } + }() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "config", + APIsDirKey: "api", + CRDsDirKey: filepath.Join("config", "crds"), + }, + OutputDir: outputDir, + } + csvVersion := "0.0.1" + g := NewCSV(cfg, csvVersion, "") + + if err := g.Generate(); err != nil { + t.Fatalf("Failed to execute CSV generator: %v", err) + } + + csvFileName := getCSVFileName(testProjectName, csvVersion) + + // Read expected CSV + expCatalogDir := filepath.Join("expected-catalog", OLMCatalogChildDir) + csvExpBytes, err := ioutil.ReadFile(filepath.Join(expCatalogDir, testProjectName, csvVersion, csvFileName)) + if err != nil { + t.Fatalf("Failed to read expected CSV file: %v", err) + } + csvExp := string(csvExpBytes) + + // Read generated CSV from OutputDir/olm-catalog + outputCatalogDir := filepath.Join(cfg.OutputDir, OLMCatalogChildDir) + csvOutputBytes, err := ioutil.ReadFile(filepath.Join(outputCatalogDir, testProjectName, csvVersion, csvFileName)) + if err != nil { + t.Fatalf("Failed to read output CSV file: %v", err) + } + csvOutput := string(csvOutputBytes) + + assert.Equal(t, csvExp, csvOutput) +} + +func TestUpgradeFromExistingCSVWithInputsToOutput(t *testing.T) { + // Change directory to project root so the test cases can form the correct pkg imports + cleanupFunc := chDirWithCleanup(t, testNonStandardLayoutDataDir) + defer cleanupFunc() + + // Temporary output dir for generating catalog bundle + outputDir, err := ioutil.TempDir("", t.Name()+"-output-catalog") + if err != nil { + log.Fatal(err) + } + // Clean up output catalog dir + defer func() { + if err := os.RemoveAll(outputDir); err != nil && !os.IsNotExist(err) { + // Not a test failure since files in /tmp will eventually get deleted + t.Logf("Failed to remove tmp generated catalog directory (%s): %v", outputDir, err) + } + }() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "config", + APIsDirKey: "api", + CRDsDirKey: filepath.Join("config", "crds"), + }, + OutputDir: outputDir, + } + fromVersion := "0.0.3" + csvVersion := "0.0.4" + + // Copy over expected fromVersion CSV bundle directory to the output dir + // so the test can upgrade from it + outputFromCSVDir := filepath.Join(outputDir, OLMCatalogChildDir, testProjectName) + if err := os.MkdirAll(outputFromCSVDir, os.FileMode(fileutil.DefaultDirFileMode)); err != nil { + t.Fatalf("Failed to create CSV bundle dir (%s) for fromVersion (%s): %v", outputFromCSVDir, fromVersion, err) + } + expCatalogDir := filepath.Join("expected-catalog", OLMCatalogChildDir) + expFromCSVDir := filepath.Join(expCatalogDir, testProjectName, fromVersion) + cmd := exec.Command("cp", "-r", expFromCSVDir, outputFromCSVDir) + t.Logf("Copying expected fromVersion CSV manifest dir %#v", cmd.Args) + if err := projutil.ExecCmd(cmd); err != nil { + t.Fatalf("Failed to copy expected CSV bundle dir (%s) to output dir (%s): %v", expFromCSVDir, outputFromCSVDir, err) + } + + // Upgrade new CSV from old + g := NewCSV(cfg, csvVersion, fromVersion) + if err := g.Generate(); err != nil { + t.Fatalf("Failed to execute CSV generator: %v", err) + } + csvFileName := getCSVFileName(testProjectName, csvVersion) + + // Read expected CSV + expCsvFile := filepath.Join(expCatalogDir, testProjectName, csvVersion, csvFileName) + csvExpBytes, err := ioutil.ReadFile(expCsvFile) + if err != nil { + t.Fatalf("Failed to read expected CSV file: %v", err) + } + csvExp := string(csvExpBytes) + + // Read generated CSV from OutputDir/olm-catalog + outputCatalogDir := filepath.Join(cfg.OutputDir, OLMCatalogChildDir) + csvOutputBytes, err := ioutil.ReadFile(filepath.Join(outputCatalogDir, testProjectName, csvVersion, csvFileName)) + if err != nil { + t.Fatalf("Failed to read output CSV file: %v", err) + } + csvOutput := string(csvOutputBytes) + + assert.Equal(t, csvExp, csvOutput) +} + +// TODO: This test is only updating the existing CSV +// deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml +// present in testdata/go +// Fix to generate a new CSV rather than only update an existing one +func TestGoCSVFromNew(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "deploy", + APIsDirKey: filepath.Join("pkg", "apis"), + CRDsDirKey: filepath.Join("deploy", "crds_v1beta1"), + }, + OutputDir: "deploy", + } + g := NewCSV(cfg, csvVersion, "") + fileMap, err := g.(csvGenerator).generate() + if err != nil { + t.Fatalf("Failed to execute CSV generator: %v", err) + } + + csvExpFile := getCSVFileName(testProjectName, csvVersion) + csvExpBytes, err := ioutil.ReadFile(filepath.Join(OLMCatalogDir, testProjectName, csvVersion, csvExpFile)) + if err != nil { + t.Fatalf("Failed to read expected CSV file: %v", err) + } + csvExp := string(csvExpBytes) + // Replace image tag, which is retrieved from the deployment and is + // different than that in the expected CSV, but doesn't matter for this test. + csvExp = strings.Replace(csvExp, + "image: quay.io/example/memcached-operator:v0.0.2", + "image: quay.io/example/memcached-operator:v0.0.3", + -1) + if b, ok := fileMap[csvExpFile]; !ok { + t.Errorf("Failed to generate CSV for version %s", csvVersion) + } else { + assert.Equal(t, csvExp, string(b)) + } +} + +func TestGoCSVFromOld(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "deploy", + APIsDirKey: filepath.Join("pkg", "apis"), + CRDsDirKey: filepath.Join("deploy", "crds_v1beta1"), + }, + OutputDir: "deploy", + } + g := NewCSV(cfg, csvVersion, fromVersion) + fileMap, err := g.(csvGenerator).generate() + if err != nil { + t.Fatalf("Failed to execute CSV generator: %v", err) + } + + csvExpFile := getCSVFileName(testProjectName, csvVersion) + csvExpBytes, err := ioutil.ReadFile(filepath.Join(OLMCatalogDir, testProjectName, csvVersion, csvExpFile)) + if err != nil { + t.Fatalf("Failed to read expected CSV file: %v", err) + } + csvExp := string(csvExpBytes) + if b, ok := fileMap[csvExpFile]; !ok { + t.Errorf("Failed to generate CSV for version %s", csvVersion) + } else { + assert.Equal(t, csvExp, string(b)) + } +} + +func TestGoCSVWithInvalidManifestsDir(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "notExist", + APIsDirKey: filepath.Join("pkg", "apis"), + CRDsDirKey: "notExist", + }, + OutputDir: "deploy", + } + + g := NewCSV(cfg, notExistVersion, "") + _, err := g.(csvGenerator).generate() + if err == nil { + t.Fatalf("Failed to get error for running CSV generator"+ + "on non-existent manifests directory: %s", cfg.Inputs[DeployDirKey]) + } +} + +func TestGoCSVWithEmptyManifestsDir(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "emptydir", + APIsDirKey: filepath.Join("pkg", "apis"), + CRDsDirKey: "emptydir", + }, + OutputDir: "emptydir", + } + + g := NewCSV(cfg, notExistVersion, "") + fileMap, err := g.(csvGenerator).generate() + if err != nil { + t.Fatalf("Failed to execute CSV generator: %v", err) + } + + // Create an empty CSV. + csv, err := newCSV(testProjectName, notExistVersion) + if err != nil { + t.Fatal(err) + } + csvExpBytes, err := internalk8sutil.GetObjectBytes(csv, yaml.Marshal) + if err != nil { + t.Fatal(err) + } + csvExpFile := getCSVFileName(testProjectName, notExistVersion) + if b, ok := fileMap[csvExpFile]; !ok { + t.Errorf("Failed to generate CSV for version %s", notExistVersion) + } else { + assert.Equal(t, string(csvExpBytes), string(b)) + } +} + +func TestUpdateVersion(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + csv, err := getCSVFromDir(filepath.Join(OLMCatalogDir, testProjectName, fromVersion)) + if err != nil { + t.Fatal("Failed to get new CSV") + } + + cfg := gen.Config{ + OperatorName: testProjectName, + Inputs: map[string]string{ + DeployDirKey: "deploy", + APIsDirKey: filepath.Join("pkg", "apis"), + CRDsDirKey: filepath.Join("deploy", "crds_v1beta1"), + }, + OutputDir: "deploy", + } + g := NewCSV(cfg, csvVersion, fromVersion) + if err := g.(csvGenerator).updateCSVVersions(csv); err != nil { + t.Fatalf("Failed to update csv with version %s: (%v)", csvVersion, err) + } + + wantedSemver, err := semver.Parse(csvVersion) + if err != nil { + t.Errorf("Failed to parse %s: %v", csvVersion, err) + } + if !csv.Spec.Version.Equals(wantedSemver) { + t.Errorf("Wanted csv version %v, got %v", wantedSemver, csv.Spec.Version) + } + wantedName := getCSVName(testProjectName, csvVersion) + if csv.ObjectMeta.Name != wantedName { + t.Errorf("Wanted csv name %s, got %s", wantedName, csv.ObjectMeta.Name) + } + + csvDepSpecs := csv.Spec.InstallStrategy.StrategySpec.DeploymentSpecs + if len(csvDepSpecs) != 1 { + t.Fatal("No deployment specs in CSV") + } + csvPodImage := csvDepSpecs[0].Spec.Template.Spec.Containers[0].Image + if len(csvDepSpecs[0].Spec.Template.Spec.Containers) != 1 { + t.Fatal("No containers in CSV deployment spec") + } + // updateCSVVersions should not update podspec image. + wantedImage := "quay.io/example/memcached-operator:v0.0.2" + if csvPodImage != wantedImage { + t.Errorf("Podspec image changed from %s to %s", wantedImage, csvPodImage) + } + + wantedReplaces := getCSVName(testProjectName, fromVersion) + if csv.Spec.Replaces != wantedReplaces { + t.Errorf("Wanted csv replaces %s, got %s", wantedReplaces, csv.Spec.Replaces) + } +} + +func TestSetAndCheckOLMNamespaces(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + depBytes, err := ioutil.ReadFile(filepath.Join(scaffold.DeployDir, "operator.yaml")) + if err != nil { + t.Fatalf("Failed to read Deployment bytes: %v", err) + } + + // The test operator.yaml doesn't have "olm.targetNamespaces", so first + // check that depHasOLMNamespaces() returns false. + dep := appsv1.Deployment{} + if err := yaml.Unmarshal(depBytes, &dep); err != nil { + t.Fatalf("Failed to unmarshal Deployment bytes: %v", err) + } + if depHasOLMNamespaces(dep) { + t.Error("Expected depHasOLMNamespaces to return false, got true") + } + + // Insert "olm.targetNamespaces" into WATCH_NAMESPACE and check that + // depHasOLMNamespaces() returns true. + setWatchNamespacesEnv(&dep) + if !depHasOLMNamespaces(dep) { + t.Error("Expected depHasOLMNamespaces to return true, got false") + } + + // Overwrite WATCH_NAMESPACE and check that depHasOLMNamespaces() returns + // false. + overwriteContainerEnvVar(&dep, k8sutil.WatchNamespaceEnvVar, newEnvVar("FOO", "bar")) + if depHasOLMNamespaces(dep) { + t.Error("Expected depHasOLMNamespaces to return false, got true") + } + + // Insert "olm.targetNamespaces" elsewhere in the deployment pod spec + // and check that depHasOLMNamespaces() returns true. + dep = appsv1.Deployment{} + if err := yaml.Unmarshal(depBytes, &dep); err != nil { + t.Fatalf("Failed to unmarshal Deployment bytes: %v", err) + } + dep.Spec.Template.ObjectMeta.Labels["namespace"] = olmTNMeta + if !depHasOLMNamespaces(dep) { + t.Error("Expected depHasOLMNamespaces to return true, got false") + } +} diff --git a/internal/scaffold/olm-catalog/csv_updaters.go b/internal/generate/olm-catalog/csv_updaters.go similarity index 74% rename from internal/scaffold/olm-catalog/csv_updaters.go rename to internal/generate/olm-catalog/csv_updaters.go index da9cb3a5180..1f425360855 100644 --- a/internal/scaffold/olm-catalog/csv_updaters.go +++ b/internal/generate/olm-catalog/csv_updaters.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package catalog +package olmcatalog import ( "bytes" @@ -20,9 +20,10 @@ import ( goerrors "errors" "fmt" "sort" + "strings" - "github.com/operator-framework/operator-sdk/internal/scaffold" - "github.com/operator-framework/operator-sdk/internal/scaffold/olm-catalog/descriptor" + "github.com/operator-framework/operator-registry/pkg/registry" + "github.com/operator-framework/operator-sdk/internal/generate/olm-catalog/descriptor" "github.com/operator-framework/operator-sdk/pkg/k8sutil" "github.com/ghodss/yaml" @@ -199,39 +200,78 @@ var _ csvUpdater = crds{} // apply updates csv's "owned" CRDDescriptions. "required" CRDDescriptions are // left as-is, since they are user-defined values. -// apply will only make a new spec.customresourcedefinitions.owned element for -// a type if an annotation is present on that type's declaration. func (us crds) apply(csv *olmapiv1alpha1.ClusterServiceVersion) error { - ownedCRDs := []olmapiv1alpha1.CRDDescription{} + ownedDescs := []olmapiv1alpha1.CRDDescription{} + descMap := map[registry.DefinitionKey]olmapiv1alpha1.CRDDescription{} + for _, owned := range csv.Spec.CustomResourceDefinitions.Owned { + defKey := registry.DefinitionKey{ + Name: owned.Name, + Version: owned.Version, + Kind: owned.Kind, + } + descMap[defKey] = owned + } for _, u := range us { crd := apiextv1beta1.CustomResourceDefinition{} if err := yaml.Unmarshal(u, &crd); err != nil { return err } for _, ver := range crd.Spec.Versions { - // Parse CRD descriptors from source code comments and annotations. - gvk := schema.GroupVersionKind{ - Group: crd.Spec.Group, + defKey := registry.DefinitionKey{ + Name: crd.GetName(), Version: ver.Name, Kind: crd.Spec.Names.Kind, } - newCRDDesc, err := descriptor.GetCRDDescriptionForGVK(scaffold.ApisDir, gvk) - if err != nil { - if goerrors.Is(err, descriptor.ErrAPIDirNotExist) { - log.Infof("Directory for API %s does not exist. Skipping CSV annotation parsing for API.", gvk) - } else if goerrors.Is(err, descriptor.ErrAPITypeNotFound) { - log.Infof("No kind type found for API %s. Skipping CSV annotation parsing for API.", gvk) - } else { - return fmt.Errorf("failed to set CRD descriptors for %s: %v", gvk, err) - } - continue + if owned, ownedExists := descMap[defKey]; ownedExists { + ownedDescs = append(ownedDescs, owned) + } else { + ownedDescs = append(ownedDescs, olmapiv1alpha1.CRDDescription{ + Name: defKey.Name, + Version: defKey.Version, + Kind: defKey.Kind, + }) + } + } + } + csv.Spec.CustomResourceDefinitions.Owned = ownedDescs + sort.Sort(descSorter(csv.Spec.CustomResourceDefinitions.Owned)) + sort.Sort(descSorter(csv.Spec.CustomResourceDefinitions.Required)) + return nil +} + +func updateDescriptions(csv *olmapiv1alpha1.ClusterServiceVersion, searchDir string) error { + updatedDescriptions := []olmapiv1alpha1.CRDDescription{} + for _, currDescription := range csv.Spec.CustomResourceDefinitions.Owned { + group := currDescription.Name + if split := strings.Split(currDescription.Name, "."); len(split) > 1 { + group = strings.Join(split[1:], ".") + } + // Parse CRD descriptors from source code comments and annotations. + gvk := schema.GroupVersionKind{ + Group: group, + Version: currDescription.Version, + Kind: currDescription.Kind, + } + newDescription, err := descriptor.GetCRDDescriptionForGVK(searchDir, gvk) + if err != nil { + if goerrors.Is(err, descriptor.ErrAPIDirNotExist) { + log.Infof("Directory for API %s does not exist. Skipping CSV annotation parsing for API.", gvk) + } else if goerrors.Is(err, descriptor.ErrAPITypeNotFound) { + log.Infof("No kind type found for API %s. Skipping CSV annotation parsing for API.", gvk) + } else { + // TODO: Should we ignore all CSV annotation parsing errors and simply log the error + // like we do for the above cases. + return fmt.Errorf("failed to set CRD descriptors for %s: %v", gvk, err) } - // Only set the name if no error was returned. - newCRDDesc.Name = crd.GetName() - ownedCRDs = append(ownedCRDs, newCRDDesc) + // Keep the existing description and don't update on error + updatedDescriptions = append(updatedDescriptions, currDescription) + } else { + // Replace the existing description with the newly parsed one + newDescription.Name = currDescription.Name + updatedDescriptions = append(updatedDescriptions, newDescription) } } - csv.Spec.CustomResourceDefinitions.Owned = ownedCRDs + csv.Spec.CustomResourceDefinitions.Owned = updatedDescriptions sort.Sort(descSorter(csv.Spec.CustomResourceDefinitions.Owned)) sort.Sort(descSorter(csv.Spec.CustomResourceDefinitions.Required)) return nil diff --git a/internal/scaffold/olm-catalog/descriptor/descriptor.go b/internal/generate/olm-catalog/descriptor/descriptor.go similarity index 54% rename from internal/scaffold/olm-catalog/descriptor/descriptor.go rename to internal/generate/olm-catalog/descriptor/descriptor.go index 4a8380407ae..4b2ea6e9a36 100644 --- a/internal/scaffold/olm-catalog/descriptor/descriptor.go +++ b/internal/generate/olm-catalog/descriptor/descriptor.go @@ -18,13 +18,11 @@ import ( "errors" "fmt" "os" - "path" "path/filepath" "strings" - "github.com/operator-framework/operator-sdk/internal/util/projutil" - olmapiv1alpha1 "github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1" + log "github.com/sirupsen/logrus" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/gengo/parser" "k8s.io/gengo/types" @@ -51,19 +49,55 @@ func GetCRDDescriptionForGVK(apisDir string, gvk schema.GroupVersionKind) (olmap if strings.Contains(group, ".") { group = strings.Split(group, ".")[0] } - apiDir := filepath.Join(apisDir, group, gvk.Version) - universe, err := getTypesFromDir(apiDir) + + // Check if apisDir exists + exists, err := isDirExist(apisDir) if err != nil { - if os.IsNotExist(err) { - return olmapiv1alpha1.CRDDescription{}, ErrAPIDirNotExist - } return olmapiv1alpha1.CRDDescription{}, err } - apiPkg := path.Join(projutil.GetGoPkg(), filepath.ToSlash(apiDir)) - pkgTypes, err := getTypesForPkg(apiPkg, universe) + if !exists { + log.Debugf("Could not find API types directory: %s", apisDir) + return olmapiv1alpha1.CRDDescription{}, ErrAPIDirNotExist + } + + // Check if the kind pkg is at the expected layout + // multi-group layout: // + // single-group layout: / + expectedPkgPath, err := getExpectedPkgLayout(apisDir, group, gvk.Version) if err != nil { return olmapiv1alpha1.CRDDescription{}, err } + + // Get pkg types for the given GVK + var pkgTypes []*types.Type + if expectedPkgPath != "" { + // Look for the pkg types at the expected single or multi group import path + universe, err := getPkgsFromDirRecursive(expectedPkgPath) + if err != nil { + return olmapiv1alpha1.CRDDescription{}, err + } + pkgTypes, err = getTypesForPkgPath(expectedPkgPath, universe) + if err != nil { + return olmapiv1alpha1.CRDDescription{}, err + } + } else { + // Unknown apis directory layout: /.../ + // Look in recursively for expected pkg name + + // TODO: gengo.parse.AddDirRecursive() will (sometimes?) fail if the + // root apisDir has no .go files. + // Workaround for this is to have a doc.go file in the package. + // Move away from using gengo in the future if possible. + universe, err := getPkgsFromDirRecursive(apisDir) + if err != nil { + return olmapiv1alpha1.CRDDescription{}, err + } + pkgTypes, err = getTypesForPkgName(gvk.Version, universe) + if err != nil { + return olmapiv1alpha1.CRDDescription{}, err + } + } + kindType := findKindType(gvk.Kind, pkgTypes) if kindType == nil { return olmapiv1alpha1.CRDDescription{}, ErrAPITypeNotFound @@ -119,15 +153,59 @@ func GetCRDDescriptionForGVK(apisDir string, gvk schema.GroupVersionKind) (olmap return crdDesc, nil } -// getTypesFromDir gets all Go types from dir. -func getTypesFromDir(dir string) (types.Universe, error) { +func isDirExist(path string) (bool, error) { + fileInfo, err := os.Stat(path) + if err != nil { + if os.IsNotExist(err) { + return false, nil + } + return false, err + } + return fileInfo.IsDir(), nil +} + +// getExpectedPkgLayout checks the directory layout in apisDir +// for single and multi group layouts and returns the expected pkg path +// for the group and version. +// Returns empty string if neither single or multi group layout is detected +// multi group path: // +// single group path: / +func getExpectedPkgLayout(apisDir, group, version string) (expectedPkgPath string, err error) { + groupVersionDir := filepath.Join(apisDir, group, version) + if isMultiGroupLayout, err := isDirExist(groupVersionDir); isMultiGroupLayout { + if err != nil { + return "", err + } + return groupVersionDir, nil + } + versionDir := filepath.Join(apisDir, version) + if isSingleGroupLayout, err := isDirExist(versionDir); isSingleGroupLayout { + if err != nil { + return "", err + } + return versionDir, nil + } + // Neither multi nor single group layout + return "", nil +} + +// getPkgsFromDirRecursive gets all Go types from dir and recursively its sub directories. +// dir must be the project relative path to the pkg directory +func getPkgsFromDirRecursive(dir string) (types.Universe, error) { if _, err := os.Stat(dir); err != nil { return nil, err } + p := parser.New() + // Gengo's AddDirRecursive fails to load subdir pkgs if the root dir + // isn't the full pkg import path, or begins with ./ + // Use path relative to current dir + // TODO: Turn abs path into ./... relative path as well if !filepath.IsAbs(dir) && !strings.HasPrefix(dir, ".") { dir = fmt.Sprintf(".%s%s", string(filepath.Separator), dir) } - p := parser.New() + // TODO(hasbro17): AddDirRecursive can be noisy with klog warnings + // when it skips directories with no .go files. + // Silence those warnings unless in debug mode. if err := p.AddDirRecursive(dir); err != nil { return nil, err } @@ -138,10 +216,11 @@ func getTypesFromDir(dir string) (types.Universe, error) { return universe, nil } -func getTypesForPkg(pkgPath string, universe types.Universe) (pkgTypes []*types.Type, err error) { +// getTypesForPkgPath find the pkg with the given path in universe +func getTypesForPkgPath(pkgPath string, universe types.Universe) (pkgTypes []*types.Type, err error) { var pkg *types.Package for _, upkg := range universe { - if strings.HasPrefix(upkg.Path, pkgPath) || strings.HasPrefix(upkg.Path, "."+string(filepath.Separator)) { + if strings.HasSuffix(upkg.Path, pkgPath) { pkg = upkg break } @@ -155,6 +234,23 @@ func getTypesForPkg(pkgPath string, universe types.Universe) (pkgTypes []*types. return pkgTypes, nil } +func getTypesForPkgName(pkgName string, universe types.Universe) (pkgTypes []*types.Type, err error) { + var pkg *types.Package + for _, upkg := range universe { + if upkg.Name == pkgName { + pkg = upkg + break + } + } + if pkg == nil { + return nil, fmt.Errorf("no package found for %s", pkgName) + } + for _, t := range pkg.Types { + pkgTypes = append(pkgTypes, t) + } + return pkgTypes, nil +} + func findKindType(kind string, pkgTypes []*types.Type) *types.Type { for _, t := range pkgTypes { if t.Name.Name == kind { diff --git a/internal/scaffold/olm-catalog/descriptor/descriptor_test.go b/internal/generate/olm-catalog/descriptor/descriptor_test.go similarity index 61% rename from internal/scaffold/olm-catalog/descriptor/descriptor_test.go rename to internal/generate/olm-catalog/descriptor/descriptor_test.go index 07744be74f7..4f8c4633866 100644 --- a/internal/scaffold/olm-catalog/descriptor/descriptor_test.go +++ b/internal/generate/olm-catalog/descriptor/descriptor_test.go @@ -26,48 +26,86 @@ import ( "github.com/ghodss/yaml" olmapiv1alpha1 "github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/gengo/types" ) -const testFrameworkPackage = "github.com/operator-framework/operator-sdk/test/test-framework" - -func getTestFrameworkDir(t *testing.T) string { - t.Helper() - absPath, err := os.Getwd() - if err != nil { - t.Fatal(err) - } - sdkPath := absPath[:strings.Index(absPath, "internal")] - tfDir := filepath.Join(sdkPath, "test", "test-framework") - // parser.AddDirRecursive doesn't like absolute paths. - relPath, err := filepath.Rel(absPath, tfDir) - if err != nil { - t.Fatal(err) - } - return relPath -} +var ( + testDataDir = filepath.Join("..", "..", "testdata", "go") +) func TestGetKindTypeForAPI(t *testing.T) { - cases := []struct { + multiAPIRootDir := filepath.Join("pkg", "apis") + singleAPIRootDir := "api" + group := "cache" + version := "v1alpha1" + + subTests := []struct { description string - pkg, kind string - numPkgTypes int - wantNil bool + // path to apis types root dir e.g pkg/apis + apisDir string + // path to kind api pkg e.g pkg/apis/cache/v1alpha1 + expectedPkgPath string + group string + version string + kind string + numPkgTypes int + wantNil bool + // True if apis dir in the expected single or multi group layout + isExpectedLayout bool }{ { - "Find types successfully", - testFrameworkPackage, "Dummy", 22, false, + "Must Succeed: Find types for Kind from multi APIs root directory", + multiAPIRootDir, + filepath.Join(multiAPIRootDir, group, version), + group, + version, + "Dummy", + 22, + false, + true, + }, + { + "Must Fail: Find types for non-existing Kind from multi APIs root directory", + multiAPIRootDir, + filepath.Join(multiAPIRootDir, group, version), + group, + version, + "NotFound", + 22, + true, + true, }, { - "Find types with error from wrong kind", - testFrameworkPackage, "NotFound", 22, true, + "Must Succeed: Find types for Kind from single APIs root directory", + singleAPIRootDir, + filepath.Join(singleAPIRootDir, version), + group, + version, + "Memcached", + 4, + false, + true, + }, + { + "Must Fail: Find types for non-existing Kind from single APIs root directory", + singleAPIRootDir, + filepath.Join(singleAPIRootDir, version), + group, + version, + "NotFound", + 4, + true, + true, }, + // TODO: Add cases for non-standard api dir layouts: pkg/apis///version } + + // Change directory to test data dir so the test cases can form the correct pkg imports wd, err := os.Getwd() if err != nil { t.Fatal(err) } - tfDir := getTestFrameworkDir(t) - if err := os.Chdir(tfDir); err != nil { + if err := os.Chdir(testDataDir); err != nil { t.Fatal(err) } defer func() { @@ -75,40 +113,65 @@ func TestGetKindTypeForAPI(t *testing.T) { t.Fatal(err) } }() - tfAPIDir := filepath.Join("pkg", "apis", "cache", "v1alpha1") - universe, err := getTypesFromDir(tfAPIDir) - if err != nil { - t.Fatal(err) - } - for _, c := range cases { - pkgTypes, err := getTypesForPkg(c.pkg, universe) - if err != nil { - t.Fatal(err) - } - if n := len(pkgTypes); n != c.numPkgTypes { - t.Errorf("%s: expected %d package types, got %d", c.description, c.numPkgTypes, n) - } - kindType := findKindType(c.kind, pkgTypes) - if c.wantNil && kindType != nil { - t.Errorf("%s: expected type %q to not be found", c.description, kindType.Name) - } - if !c.wantNil && kindType == nil { - t.Errorf("%s: expected type %q to be found", c.description, c.kind) - } - if !c.wantNil && kindType != nil && kindType.Name.Name != c.kind { - t.Errorf("%s: expected type %q to have type name %q", c.description, kindType.Name, c.kind) - } + for _, st := range subTests { + t.Run(st.description, func(t *testing.T) { + expectedPkgPath, err := getExpectedPkgLayout(st.apisDir, st.group, st.version) + if err != nil { + t.Fatalf("Failed to getExpectedPkgLayout(%s, %s, %s): %v", st.apisDir, st.group, st.version, err) + } + if st.isExpectedLayout { + if expectedPkgPath == "" || !strings.HasSuffix(expectedPkgPath, st.expectedPkgPath) { + t.Fatalf("Expected (%s) as suffix to expected pkg path (%s)", st.expectedPkgPath, expectedPkgPath) + } + } + + var pkgTypes []*types.Type + if st.isExpectedLayout { + universe, err := getPkgsFromDirRecursive(expectedPkgPath) + if err != nil { + t.Fatalf("Failed to get universe of types from API root directory (%s): %v)", st.apisDir, err) + } + pkgTypes, err = getTypesForPkgPath(expectedPkgPath, universe) + if err != nil { + t.Fatalf("Failed to get types of pkg path (%s) from API root directory(%s): %v)", + expectedPkgPath, st.apisDir, err) + } + } else { + universe, err := getPkgsFromDirRecursive(st.apisDir) + if err != nil { + t.Fatalf("Failed to get universe of types from API root directory (%s): %v)", st.apisDir, err) + } + pkgTypes, err = getTypesForPkgName(st.version, universe) + if err != nil { + t.Fatalf("Failed to get types of pkg name (%s) from API root directory(%s): %v)", st.version, st.apisDir, err) + } + } + + if n := len(pkgTypes); n != st.numPkgTypes { + t.Errorf("Expected %d package types, got %d", st.numPkgTypes, n) + } + kindType := findKindType(st.kind, pkgTypes) + if st.wantNil && kindType != nil { + t.Errorf("Expected type %q to not be found", kindType.Name) + } + if !st.wantNil && kindType == nil { + t.Errorf("Expected type %q to be found", st.kind) + } + if !st.wantNil && kindType != nil && kindType.Name.Name != st.kind { + t.Errorf("Expected type %q to have type name %q", kindType.Name, st.kind) + } + }) } } func TestGetCRDDescriptionForGVK(t *testing.T) { + wd, err := os.Getwd() if err != nil { t.Fatal(err) } - tfDir := getTestFrameworkDir(t) - if err := os.Chdir(tfDir); err != nil { + if err := os.Chdir(testDataDir); err != nil { t.Fatal(err) } defer func() { @@ -123,6 +186,7 @@ func TestGetCRDDescriptionForGVK(t *testing.T) { return xdescs } + // TODO(hasbro17): Change to run as subtests cases := []struct { description string apisDir string diff --git a/internal/scaffold/olm-catalog/descriptor/parse.go b/internal/generate/olm-catalog/descriptor/parse.go similarity index 100% rename from internal/scaffold/olm-catalog/descriptor/parse.go rename to internal/generate/olm-catalog/descriptor/parse.go diff --git a/internal/scaffold/olm-catalog/descriptor/parse_test.go b/internal/generate/olm-catalog/descriptor/parse_test.go similarity index 100% rename from internal/scaffold/olm-catalog/descriptor/parse_test.go rename to internal/generate/olm-catalog/descriptor/parse_test.go diff --git a/internal/scaffold/olm-catalog/descriptor/search.go b/internal/generate/olm-catalog/descriptor/search.go similarity index 100% rename from internal/scaffold/olm-catalog/descriptor/search.go rename to internal/generate/olm-catalog/descriptor/search.go diff --git a/internal/generate/olm-catalog/package_manifest.go b/internal/generate/olm-catalog/package_manifest.go index ae3c0aecda3..bb65626954d 100644 --- a/internal/generate/olm-catalog/package_manifest.go +++ b/internal/generate/olm-catalog/package_manifest.go @@ -26,6 +26,7 @@ import ( "github.com/operator-framework/api/pkg/validation" "github.com/operator-framework/operator-registry/pkg/registry" "github.com/operator-framework/operator-sdk/internal/generate/gen" + "github.com/operator-framework/operator-sdk/internal/scaffold" "github.com/operator-framework/operator-sdk/internal/util/fileutil" "github.com/ghodss/yaml" @@ -33,9 +34,7 @@ import ( ) const ( - PackageManifestFileExt = ".package.yaml" - - ManifestsDirKey = "manifests" + packageManifestFileExt = ".package.yaml" ) type pkgGenerator struct { @@ -60,21 +59,33 @@ func NewPackageManifest(cfg gen.Config, csvVersion, channel string, isDefault bo channelIsDefault: isDefault, fileName: getPkgFileName(cfg.OperatorName), } - if g.Inputs == nil { - g.Inputs = map[string]string{} - } - if manifests, ok := g.Inputs[ManifestsDirKey]; !ok || manifests == "" { - g.Inputs[ManifestsDirKey] = filepath.Join(OLMCatalogDir, g.OperatorName) - } + + // Pkg manifest generator has no defined inputs + g.Inputs = map[string]string{} + + // The olm-catalog directory location depends on where the output directory is set. if g.OutputDir == "" { - g.OutputDir = filepath.Join(OLMCatalogDir, g.OperatorName) + g.OutputDir = scaffold.DeployDir } + return g } +func isFileExist(path string) bool { + _, err := os.Stat(path) + if err != nil { + if os.IsNotExist(err) { + return false + } + // TODO: return and handle this error + log.Fatalf("Failed to stat %s: %v", path, err) + } + return true +} + // getPkgFileName will return the name of the PackageManifestFile func getPkgFileName(operatorName string) string { - return strings.ToLower(operatorName) + PackageManifestFileExt + return strings.ToLower(operatorName) + packageManifestFileExt } func (g pkgGenerator) Generate() error { @@ -85,11 +96,13 @@ func (g pkgGenerator) Generate() error { if len(fileMap) == 0 { return errors.New("error generating package manifest: no generated file found") } - if err = os.MkdirAll(g.OutputDir, fileutil.DefaultDirFileMode); err != nil { - return fmt.Errorf("error mkdir %s: %v", g.OutputDir, err) + pkgManifestOutputDir := filepath.Join(g.OutputDir, OLMCatalogChildDir, g.OperatorName) + if err = os.MkdirAll(pkgManifestOutputDir, fileutil.DefaultDirFileMode); err != nil { + return fmt.Errorf("error mkdir %s: %v", pkgManifestOutputDir, err) } for fileName, b := range fileMap { - path := filepath.Join(g.OutputDir, fileName) + path := filepath.Join(pkgManifestOutputDir, fileName) + log.Debugf("Package manifest generator writing %s", path) if err = ioutil.WriteFile(path, b, fileutil.DefaultFileMode); err != nil { return err } @@ -124,23 +137,22 @@ func (g pkgGenerator) generate() (map[string][]byte, error) { return fileMap, nil } -// buildPackageManifest will create a registry.PackageManifest from scratch, or modify +// buildPackageManifest will create a registry.PackageManifest from scratch, or reads // an existing one if found at the expected path. func (g pkgGenerator) buildPackageManifest() (registry.PackageManifest, error) { pkg := registry.PackageManifest{} - path := filepath.Join(g.Inputs[ManifestsDirKey], g.fileName) - if _, err := os.Stat(path); err == nil { - b, err := ioutil.ReadFile(path) + olmCatalogDir := filepath.Join(g.OutputDir, OLMCatalogChildDir) + existingPkgManifest := filepath.Join(olmCatalogDir, g.OperatorName, g.fileName) + if isFileExist(existingPkgManifest) { + b, err := ioutil.ReadFile(existingPkgManifest) if err != nil { - return pkg, fmt.Errorf("failed to read package manifest %s: %v", path, err) + return pkg, fmt.Errorf("failed to read package manifest %s: %v", existingPkgManifest, err) } if err = yaml.Unmarshal(b, &pkg); err != nil { - return pkg, fmt.Errorf("failed to unmarshal package manifest %s: %v", path, err) + return pkg, fmt.Errorf("failed to unmarshal package manifest %s: %v", existingPkgManifest, err) } - } else if os.IsNotExist(err) { - pkg = newPackageManifest(g.OperatorName, g.channel, g.csvVersion) } else { - return pkg, fmt.Errorf("error reading package manifest %s: %v", path, err) + pkg = newPackageManifest(g.OperatorName, g.channel, g.csvVersion) } return pkg, nil } diff --git a/internal/generate/olm-catalog/package_manifest_test.go b/internal/generate/olm-catalog/package_manifest_test.go index 438f4125d02..466b2df724e 100644 --- a/internal/generate/olm-catalog/package_manifest_test.go +++ b/internal/generate/olm-catalog/package_manifest_test.go @@ -15,6 +15,9 @@ package olmcatalog import ( + "io/ioutil" + "log" + "os" "path/filepath" "reflect" "testing" @@ -25,29 +28,64 @@ import ( "github.com/stretchr/testify/assert" ) -const ( - testProjectName = "memcached-operator" - csvVersion = "0.0.3" -) +func TestGeneratePkgManifestToOutput(t *testing.T) { + cleanupFunc := chDirWithCleanup(t, testNonStandardLayoutDataDir) + defer cleanupFunc() -var ( - testDataDir = filepath.Join("..", "testdata") - testGoDataDir = filepath.Join(testDataDir, "go") -) + // Temporary output dir for generating catalog bundle + outputDir, err := ioutil.TempDir("", t.Name()+"-output-catalog") + if err != nil { + log.Fatal(err) + } + // Clean up output catalog dir + defer func() { + if err := os.RemoveAll(outputDir); err != nil && !os.IsNotExist(err) { + // Not a test failure since files in /tmp will eventually get deleted + t.Logf("Failed to remove tmp generated catalog directory (%s): %v", outputDir, err) + } + }() -// newTestPackageManifestGenerator returns a package manifest Generator populated with test values. -func newTestPackageManifestGenerator() gen.Generator { - inputDir := filepath.Join(testGoDataDir, OLMCatalogDir, testProjectName) cfg := gen.Config{ OperatorName: testProjectName, - Inputs: map[string]string{ManifestsDirKey: inputDir}, + OutputDir: outputDir, } + g := NewPackageManifest(cfg, csvVersion, "stable", true) - return g + if err := g.Generate(); err != nil { + t.Fatalf("Failed to execute package manifest generator: %v", err) + } + + pkgManFileName := getPkgFileName(testProjectName) + + // Read expected Package Manifest + expCatalogDir := filepath.Join("expected-catalog", OLMCatalogChildDir) + pkgManExpBytes, err := ioutil.ReadFile(filepath.Join(expCatalogDir, testProjectName, pkgManFileName)) + if err != nil { + t.Fatalf("Failed to read expected package manifest file: %v", err) + } + pkgManExp := string(pkgManExpBytes) + + // Read generated Package Manifest from OutputDir/olm-catalog + outputCatalogDir := filepath.Join(cfg.OutputDir, OLMCatalogChildDir) + pkgManOutputBytes, err := ioutil.ReadFile(filepath.Join(outputCatalogDir, testProjectName, pkgManFileName)) + if err != nil { + t.Fatalf("Failed to read output package manifest file: %v", err) + } + pkgManOutput := string(pkgManOutputBytes) + + assert.Equal(t, pkgManExp, pkgManOutput) + } func TestGeneratePackageManifest(t *testing.T) { - g := newTestPackageManifestGenerator() + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + OutputDir: "deploy", + } + g := NewPackageManifest(cfg, csvVersion, "stable", true) fileMap, err := g.(pkgGenerator).generate() if err != nil { t.Fatalf("Failed to execute package manifest generator: %v", err) @@ -61,7 +99,14 @@ func TestGeneratePackageManifest(t *testing.T) { } func TestValidatePackageManifest(t *testing.T) { - g := newTestPackageManifestGenerator() + cleanupFunc := chDirWithCleanup(t, testGoDataDir) + defer cleanupFunc() + + cfg := gen.Config{ + OperatorName: testProjectName, + OutputDir: "deploy", + } + g := NewPackageManifest(cfg, csvVersion, "stable", true) // pkg is a basic, valid package manifest. pkg, err := g.(pkgGenerator).buildPackageManifest() diff --git a/internal/generate/testdata/go/api/v1alpha1/memcached_types.go b/internal/generate/testdata/go/api/v1alpha1/memcached_types.go new file mode 100644 index 00000000000..985292f393e --- /dev/null +++ b/internal/generate/testdata/go/api/v1alpha1/memcached_types.go @@ -0,0 +1,57 @@ +// Copyright 2020 The Operator-SDK Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// MemcachedSpec defines the desired state of Memcached +type MemcachedSpec struct { + // Size is the size of the memcached deployment + // +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true + Size int32 `json:"size"` +} + +// MemcachedStatus defines the observed state of Memcached +type MemcachedStatus struct { + // Nodes are the names of the memcached pods + // +operator-sdk:gen-csv:customresourcedefinitions.statusDescriptors=true + Nodes []string `json:"nodes"` +} + +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object + +// Memcached is the Schema for the memcacheds API +// +kubebuilder:subresource:status +// +kubebuilder:resource:path=memcacheds,scope=Namespaced +// +kubebuilder:storageversion +// +operator-sdk:gen-csv:customresourcedefinitions.displayName="Memcached App" +type Memcached struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec MemcachedSpec `json:"spec,omitempty"` + Status MemcachedStatus `json:"status,omitempty"` +} + +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object + +// MemcachedList contains a list of Memcached +type MemcachedList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []Memcached `json:"items"` +} diff --git a/internal/generate/testdata/go/build/Dockerfile b/internal/generate/testdata/go/emptydir/.keep similarity index 100% rename from internal/generate/testdata/go/build/Dockerfile rename to internal/generate/testdata/go/emptydir/.keep diff --git a/test/test-framework/pkg/apis/cache/v1alpha1/dummy_types.go b/internal/generate/testdata/go/pkg/apis/cache/v1alpha1/dummy_types.go similarity index 99% rename from test/test-framework/pkg/apis/cache/v1alpha1/dummy_types.go rename to internal/generate/testdata/go/pkg/apis/cache/v1alpha1/dummy_types.go index b255b336ad8..e6ee1101a7c 100644 --- a/test/test-framework/pkg/apis/cache/v1alpha1/dummy_types.go +++ b/internal/generate/testdata/go/pkg/apis/cache/v1alpha1/dummy_types.go @@ -1,4 +1,4 @@ -// Copyright 2018 The Operator-SDK Authors +// Copyright 2020 The Operator-SDK Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. diff --git a/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/doc.go b/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/doc.go new file mode 100644 index 00000000000..d5094b45a66 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/doc.go @@ -0,0 +1,18 @@ +// Copyright 2020 The Operator-SDK Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package v1alpha1 contains API Schema definitions for the cache v1alpha1 API group +// +k8s:deepcopy-gen=package,register +// +groupName=cache.example.com +package v1alpha1 diff --git a/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/memcached_types.go b/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/memcached_types.go new file mode 100644 index 00000000000..156d2bb86b5 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/api/cache/v1alpha1/memcached_types.go @@ -0,0 +1,57 @@ +// Copyright 2020 The Operator-SDK Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// MemcachedSpec defines the desired state of Memcached +type MemcachedSpec struct { + // Size is the size of the memcached deployment + // +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true + Size int32 `json:"size"` +} + +// MemcachedStatus defines the observed state of Memcached +type MemcachedStatus struct { + // Nodes are the names of the memcached pods + // +operator-sdk:gen-csv:customresourcedefinitions.statusDescriptors=true + Nodes []string `json:"nodes"` +} + +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object + +// Memcached is the Schema for the memcacheds API +// +kubebuilder:subresource:status +// +kubebuilder:resource:path=memcacheds,scope=Namespaced +// +kubebuilder:storageversion +// +operator-sdk:gen-csv:customresourcedefinitions.displayName="Memcached App Display Name" +type Memcached struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec MemcachedSpec `json:"spec,omitempty"` + Status MemcachedStatus `json:"status,omitempty"` +} + +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object + +// MemcachedList contains a list of Memcached +type MemcachedList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []Memcached `json:"items"` +} diff --git a/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_memcacheds_crd.yaml b/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_memcacheds_crd.yaml new file mode 100644 index 00000000000..0109edbab9a --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_memcacheds_crd.yaml @@ -0,0 +1,57 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: memcacheds.cache.example.com +spec: + group: cache.example.com + names: + kind: Memcached + listKind: MemcachedList + plural: memcacheds + singular: memcached + scope: Namespaced + subresources: + status: {} + validation: + openAPIV3Schema: + description: Memcached is the Schema for the memcacheds API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: MemcachedSpec defines the desired state of Memcached + properties: + size: + description: Size is the size of the memcached deployment + format: int32 + type: integer + required: + - size + type: object + status: + description: MemcachedStatus defines the observed state of Memcached + properties: + nodes: + description: Nodes are the names of the memcached pods + items: + type: string + type: array + required: + - nodes + type: object + type: object + version: v1alpha1 + versions: + - name: v1alpha1 + served: true + storage: true diff --git a/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_v1alpha1_memcached_cr.yaml b/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_v1alpha1_memcached_cr.yaml new file mode 100644 index 00000000000..2b8f17c3998 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/crds/cache.example.com_v1alpha1_memcached_cr.yaml @@ -0,0 +1,7 @@ +apiVersion: cache.example.com/v1alpha1 +kind: Memcached +metadata: + name: example-memcached +spec: + # Add fields here + size: 3 diff --git a/internal/generate/testdata/non-standard-layout/config/operator.yaml b/internal/generate/testdata/non-standard-layout/config/operator.yaml new file mode 100644 index 00000000000..8a00f16ad68 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/operator.yaml @@ -0,0 +1,32 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: memcached-operator +spec: + replicas: 1 + selector: + matchLabels: + name: memcached-operator + template: + metadata: + labels: + name: memcached-operator + spec: + serviceAccountName: memcached-operator + containers: + - name: memcached-operator + image: quay.io/example/memcached-operator:v0.0.3 + command: + - memcached-operator + imagePullPolicy: Never + env: + - name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPERATOR_NAME + value: memcached-operator diff --git a/internal/generate/testdata/non-standard-layout/config/role.yaml b/internal/generate/testdata/non-standard-layout/config/role.yaml new file mode 100644 index 00000000000..d1a2a04d0d0 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/role.yaml @@ -0,0 +1,61 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: memcached-operator +rules: +- apiGroups: + - "" + resources: + - pods + - services + - services/finalizers + - endpoints + - persistentvolumeclaims + - events + - configmaps + - secrets + verbs: + - '*' +- apiGroups: + - apps + resources: + - deployments + - daemonsets + - replicasets + - statefulsets + verbs: + - '*' +- apiGroups: + - monitoring.coreos.com + resources: + - servicemonitors + verbs: + - get + - create +- apiGroups: + - apps + resourceNames: + - memcached-operator + resources: + - deployments/finalizers + verbs: + - update +- apiGroups: + - "" + resources: + - pods + verbs: + - get +- apiGroups: + - apps + resources: + - replicasets + - deployments + verbs: + - get +- apiGroups: + - cache.example.com + resources: + - '*' + verbs: + - '*' diff --git a/internal/generate/testdata/non-standard-layout/config/role_binding.yaml b/internal/generate/testdata/non-standard-layout/config/role_binding.yaml new file mode 100644 index 00000000000..322ecc9e6ac --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/role_binding.yaml @@ -0,0 +1,11 @@ +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: memcached-operator +subjects: +- kind: ServiceAccount + name: memcached-operator +roleRef: + kind: Role + name: memcached-operator + apiGroup: rbac.authorization.k8s.io diff --git a/internal/generate/testdata/non-standard-layout/config/service_account.yaml b/internal/generate/testdata/non-standard-layout/config/service_account.yaml new file mode 100644 index 00000000000..8d58bc78322 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/config/service_account.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: memcached-operator diff --git a/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml new file mode 100644 index 00000000000..65825e9192f --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml @@ -0,0 +1,153 @@ +apiVersion: operators.coreos.com/v1alpha1 +kind: ClusterServiceVersion +metadata: + annotations: + alm-examples: |- + [ + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "Memcached", + "metadata": { + "name": "example-memcached" + }, + "spec": { + "size": 3 + } + } + ] + capabilities: Basic Install + name: memcached-operator.v0.0.1 + namespace: placeholder +spec: + apiservicedefinitions: {} + customresourcedefinitions: + owned: + - description: Memcached is the Schema for the memcacheds API + displayName: Memcached App Display Name + kind: Memcached + name: memcacheds.cache.example.com + specDescriptors: + - description: Size is the size of the memcached deployment + displayName: Size + path: size + x-descriptors: + - urn:alm:descriptor:com.tectonic.ui:podCount + statusDescriptors: + - description: Nodes are the names of the memcached pods + displayName: Nodes + path: nodes + version: v1alpha1 + displayName: Memcached Operator + icon: + - base64data: "" + mediatype: "" + install: + spec: + deployments: + - name: memcached-operator + spec: + replicas: 1 + selector: + matchLabels: + name: memcached-operator + strategy: {} + template: + metadata: + labels: + name: memcached-operator + spec: + containers: + - command: + - memcached-operator + env: + - name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.annotations['olm.targetNamespaces'] + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPERATOR_NAME + value: memcached-operator + image: quay.io/example/memcached-operator:v0.0.3 + imagePullPolicy: Never + name: memcached-operator + resources: {} + serviceAccountName: memcached-operator + permissions: + - rules: + - apiGroups: + - "" + resources: + - pods + - services + - services/finalizers + - endpoints + - persistentvolumeclaims + - events + - configmaps + - secrets + verbs: + - '*' + - apiGroups: + - apps + resources: + - deployments + - daemonsets + - replicasets + - statefulsets + verbs: + - '*' + - apiGroups: + - monitoring.coreos.com + resources: + - servicemonitors + verbs: + - get + - create + - apiGroups: + - apps + resourceNames: + - memcached-operator + resources: + - deployments/finalizers + verbs: + - update + - apiGroups: + - "" + resources: + - pods + verbs: + - get + - apiGroups: + - apps + resources: + - replicasets + - deployments + verbs: + - get + - apiGroups: + - cache.example.com + resources: + - '*' + verbs: + - '*' + serviceAccountName: memcached-operator + strategy: deployment + installModes: + - supported: true + type: OwnNamespace + - supported: true + type: SingleNamespace + - supported: false + type: MultiNamespace + - supported: true + type: AllNamespaces + keywords: + - "" + maintainers: + - {} + maturity: alpha + provider: {} + version: 0.0.1 diff --git a/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml new file mode 100644 index 00000000000..3e3d7f26f8f --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml @@ -0,0 +1,154 @@ +apiVersion: operators.coreos.com/v1alpha1 +kind: ClusterServiceVersion +metadata: + annotations: + alm-examples: |- + [ + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "Memcached", + "metadata": { + "name": "example-memcached" + }, + "spec": { + "size": 3 + } + } + ] + capabilities: Basic Install + name: memcached-operator.v0.0.3 + namespace: placeholder +spec: + apiservicedefinitions: {} + customresourcedefinitions: + owned: + - description: Memcached is the Schema for the memcacheds API + displayName: Memcached App Display Name + kind: Memcached + name: memcacheds.cache.example.com + specDescriptors: + - description: Size is the size of the memcached deployment + displayName: Size + path: size + x-descriptors: + - urn:alm:descriptor:com.tectonic.ui:podCount + statusDescriptors: + - description: Nodes are the names of the memcached pods + displayName: Nodes + path: nodes + version: v1alpha1 + displayName: Memcached Operator + icon: + - base64data: "" + mediatype: "" + install: + spec: + deployments: + - name: memcached-operator + spec: + replicas: 1 + selector: + matchLabels: + name: memcached-operator + strategy: {} + template: + metadata: + labels: + name: memcached-operator + spec: + containers: + - command: + - memcached-operator + env: + - name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.annotations['olm.targetNamespaces'] + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPERATOR_NAME + value: memcached-operator + image: quay.io/example/memcached-operator:v0.0.3 + imagePullPolicy: Never + name: memcached-operator + resources: {} + serviceAccountName: memcached-operator + permissions: + - rules: + - apiGroups: + - "" + resources: + - pods + - services + - services/finalizers + - endpoints + - persistentvolumeclaims + - events + - configmaps + - secrets + verbs: + - '*' + - apiGroups: + - apps + resources: + - deployments + - daemonsets + - replicasets + - statefulsets + verbs: + - '*' + - apiGroups: + - monitoring.coreos.com + resources: + - servicemonitors + verbs: + - get + - create + - apiGroups: + - apps + resourceNames: + - memcached-operator + resources: + - deployments/finalizers + verbs: + - update + - apiGroups: + - "" + resources: + - pods + verbs: + - get + - apiGroups: + - apps + resources: + - replicasets + - deployments + verbs: + - get + - apiGroups: + - cache.example.com + resources: + - '*' + verbs: + - '*' + serviceAccountName: memcached-operator + strategy: deployment + installModes: + - supported: true + type: OwnNamespace + - supported: true + type: SingleNamespace + - supported: false + type: MultiNamespace + - supported: true + type: AllNamespaces + keywords: + - "FooBar" + - "These keywords must be preserved in the CSV update tests from 0.0.3 to 0.0.4" + maintainers: + - {} + maturity: alpha + provider: {} + version: 0.0.3 diff --git a/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.4/memcached-operator.v0.0.4.clusterserviceversion.yaml b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.4/memcached-operator.v0.0.4.clusterserviceversion.yaml new file mode 100644 index 00000000000..d40de55874c --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/0.0.4/memcached-operator.v0.0.4.clusterserviceversion.yaml @@ -0,0 +1,155 @@ +apiVersion: operators.coreos.com/v1alpha1 +kind: ClusterServiceVersion +metadata: + annotations: + alm-examples: |- + [ + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "Memcached", + "metadata": { + "name": "example-memcached" + }, + "spec": { + "size": 3 + } + } + ] + capabilities: Basic Install + name: memcached-operator.v0.0.4 + namespace: placeholder +spec: + apiservicedefinitions: {} + customresourcedefinitions: + owned: + - description: Memcached is the Schema for the memcacheds API + displayName: Memcached App Display Name + kind: Memcached + name: memcacheds.cache.example.com + specDescriptors: + - description: Size is the size of the memcached deployment + displayName: Size + path: size + x-descriptors: + - urn:alm:descriptor:com.tectonic.ui:podCount + statusDescriptors: + - description: Nodes are the names of the memcached pods + displayName: Nodes + path: nodes + version: v1alpha1 + displayName: Memcached Operator + icon: + - base64data: "" + mediatype: "" + install: + spec: + deployments: + - name: memcached-operator + spec: + replicas: 1 + selector: + matchLabels: + name: memcached-operator + strategy: {} + template: + metadata: + labels: + name: memcached-operator + spec: + containers: + - command: + - memcached-operator + env: + - name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.annotations['olm.targetNamespaces'] + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPERATOR_NAME + value: memcached-operator + image: quay.io/example/memcached-operator:v0.0.3 + imagePullPolicy: Never + name: memcached-operator + resources: {} + serviceAccountName: memcached-operator + permissions: + - rules: + - apiGroups: + - "" + resources: + - pods + - services + - services/finalizers + - endpoints + - persistentvolumeclaims + - events + - configmaps + - secrets + verbs: + - '*' + - apiGroups: + - apps + resources: + - deployments + - daemonsets + - replicasets + - statefulsets + verbs: + - '*' + - apiGroups: + - monitoring.coreos.com + resources: + - servicemonitors + verbs: + - get + - create + - apiGroups: + - apps + resourceNames: + - memcached-operator + resources: + - deployments/finalizers + verbs: + - update + - apiGroups: + - "" + resources: + - pods + verbs: + - get + - apiGroups: + - apps + resources: + - replicasets + - deployments + verbs: + - get + - apiGroups: + - cache.example.com + resources: + - '*' + verbs: + - '*' + serviceAccountName: memcached-operator + strategy: deployment + installModes: + - supported: true + type: OwnNamespace + - supported: true + type: SingleNamespace + - supported: false + type: MultiNamespace + - supported: true + type: AllNamespaces + keywords: + - FooBar + - These keywords must be preserved in the CSV update tests from 0.0.3 to 0.0.4 + maintainers: + - {} + maturity: alpha + provider: {} + replaces: memcached-operator.v0.0.3 + version: 0.0.4 diff --git a/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/memcached-operator.package.yaml b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/memcached-operator.package.yaml new file mode 100644 index 00000000000..64d34201345 --- /dev/null +++ b/internal/generate/testdata/non-standard-layout/expected-catalog/olm-catalog/memcached-operator/memcached-operator.package.yaml @@ -0,0 +1,5 @@ +channels: +- currentCSV: memcached-operator.v0.0.3 + name: stable +defaultChannel: stable +packageName: memcached-operator diff --git a/internal/scaffold/olm-catalog/config.go b/internal/scaffold/olm-catalog/config.go deleted file mode 100644 index acddbdbd5b6..00000000000 --- a/internal/scaffold/olm-catalog/config.go +++ /dev/null @@ -1,148 +0,0 @@ -// Copyright 2018 The Operator-SDK Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package catalog - -import ( - "fmt" - "io/ioutil" - "os" - "path/filepath" - - "github.com/operator-framework/operator-sdk/internal/scaffold" - - "github.com/ghodss/yaml" - log "github.com/sirupsen/logrus" -) - -// CSVConfig is a configuration file for CSV composition. Its fields contain -// file path information. -// TODO(estroz): define field for path to write CSV bundle. -// TODO(estroz): make CSVConfig a viper.Config -type CSVConfig struct { - // The operator manifest file path. Defaults to deploy/operator.yaml. - OperatorPath string `json:"operator-path,omitempty"` - // Role and ClusterRole manifest file paths. Defaults to [deploy/role.yaml]. - RolePaths []string `json:"role-paths,omitempty"` - // A list of CRD and CR manifest file paths. Defaults to [deploy/crds]. - CRDCRPaths []string `json:"crd-cr-paths,omitempty"` - // OperatorName is the name used to create the CSV and manifest file names. - // Defaults to the project's name. - OperatorName string `json:"operator-name,omitempty"` -} - -// TODO: discuss case of no config file at default path: write new file or not. -func GetCSVConfig(cfgFile string) (*CSVConfig, error) { - cfg := &CSVConfig{} - if _, err := os.Stat(cfgFile); err == nil { - cfgData, err := ioutil.ReadFile(cfgFile) - if err != nil { - return nil, err - } - if err = yaml.Unmarshal(cfgData, cfg); err != nil { - return nil, err - } - } else if !os.IsNotExist(err) { - return nil, err - } - - if err := cfg.setFields(); err != nil { - return nil, err - } - return cfg, nil -} - -const yamlExt = ".yaml" - -func (c *CSVConfig) setFields() error { - if c.OperatorPath == "" { - info, err := (&scaffold.Operator{}).GetInput() - if err != nil { - return err - } - c.OperatorPath = info.Path - } - - if len(c.RolePaths) == 0 { - info, err := (&scaffold.Role{}).GetInput() - if err != nil { - return err - } - c.RolePaths = []string{info.Path} - } - - if len(c.CRDCRPaths) == 0 { - paths, err := getManifestPathsFromDir(scaffold.CRDsDir) - if err != nil && !os.IsNotExist(err) { - return err - } - if os.IsNotExist(err) { - log.Infof("Default CRDs dir %s does not exist. Omitting field spec.customresourcedefinitions.owned"+ - " from CSV.", scaffold.CRDsDir) - } else if len(paths) == 0 { - log.Infof("Default CRDs dir %s is empty. Omitting field spec.customresourcedefinitions.owned"+ - " from CSV.", scaffold.CRDsDir) - } else { - c.CRDCRPaths = paths - } - } else { - // Allow user to specify a list of dirs to search. Avoid duplicate files. - paths, seen := make([]string, 0), make(map[string]struct{}) - for _, path := range c.CRDCRPaths { - info, err := os.Stat(path) - if err != nil { - return err - } - if info.IsDir() { - tmpPaths, err := getManifestPathsFromDir(path) - if err != nil { - return err - } - for _, p := range tmpPaths { - if _, ok := seen[p]; !ok { - paths = append(paths, p) - seen[p] = struct{}{} - } - } - } else if filepath.Ext(path) == yamlExt { - if _, ok := seen[path]; !ok { - paths = append(paths, path) - seen[path] = struct{}{} - } - } - } - c.CRDCRPaths = paths - } - - return nil -} - -func getManifestPathsFromDir(dir string) (paths []string, err error) { - err = filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { - if err != nil { - return err - } - if info == nil { - return fmt.Errorf("file info for %s was nil", path) - } - if !info.IsDir() && filepath.Ext(path) == yamlExt { - paths = append(paths, path) - } - return nil - }) - if err != nil { - return nil, err - } - return paths, nil -} diff --git a/internal/scaffold/olm-catalog/config_test.go b/internal/scaffold/olm-catalog/config_test.go deleted file mode 100644 index 4699006acaa..00000000000 --- a/internal/scaffold/olm-catalog/config_test.go +++ /dev/null @@ -1,55 +0,0 @@ -// Copyright 2018 The Operator-SDK Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package catalog - -import ( - "path/filepath" - "reflect" - "sort" - "testing" - - "github.com/operator-framework/operator-sdk/internal/scaffold" -) - -func TestConfig(t *testing.T) { - crdsDir := filepath.Join(testDataDir, scaffold.CRDsDir) - - cfg := &CSVConfig{ - CRDCRPaths: []string{crdsDir}, - } - if err := cfg.setFields(); err != nil { - t.Errorf("Set fields crd-cr paths dir only: (%v)", err) - } - if len(cfg.CRDCRPaths) != 3 { - t.Errorf("Wanted 3 crd/cr files, got: %v", cfg.CRDCRPaths) - } - - cfg = &CSVConfig{ - CRDCRPaths: []string{crdsDir, filepath.Join(crdsDir, "app.example.com_appservices_crd.yaml")}, - } - if err := cfg.setFields(); err != nil { - t.Errorf("Set fields crd-cr paths dir file mix: (%v)", err) - } - want := []string{ - filepath.Join(crdsDir, "app.example.com_v1alpha1_appservice_cr.yaml"), - filepath.Join(crdsDir, "app.example.com_appservices_crd.yaml"), - filepath.Join(crdsDir, "app.example.com_appservices2_crd.yaml"), - } - sort.Strings(want) - sort.Strings(cfg.CRDCRPaths) - if !reflect.DeepEqual(want, cfg.CRDCRPaths) { - t.Errorf("Files in crd-cr-paths do not match expected:\nwanted: %+q\ngot: %+q", want, cfg.CRDCRPaths) - } -} diff --git a/internal/scaffold/olm-catalog/csv.go b/internal/scaffold/olm-catalog/csv.go deleted file mode 100644 index c8873b19615..00000000000 --- a/internal/scaffold/olm-catalog/csv.go +++ /dev/null @@ -1,430 +0,0 @@ -// Copyright 2018 The Operator-SDK Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package catalog - -import ( - "errors" - "fmt" - "io/ioutil" - "os" - "path/filepath" - "regexp" - "sort" - "strings" - "sync" - - "github.com/operator-framework/operator-sdk/internal/scaffold" - "github.com/operator-framework/operator-sdk/internal/scaffold/input" - "github.com/operator-framework/operator-sdk/internal/util/k8sutil" - "github.com/operator-framework/operator-sdk/internal/util/projutil" - "github.com/operator-framework/operator-sdk/internal/util/yamlutil" - - "github.com/blang/semver" - "github.com/ghodss/yaml" - olmapiv1alpha1 "github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1" - olmversion "github.com/operator-framework/operator-lifecycle-manager/pkg/lib/version" - log "github.com/sirupsen/logrus" - "github.com/spf13/afero" - apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" - "k8s.io/apimachinery/pkg/runtime/schema" -) - -const ( - OLMCatalogDir = scaffold.DeployDir + string(filepath.Separator) + "olm-catalog" - CSVYamlFileExt = ".clusterserviceversion.yaml" - CSVConfigYamlFile = "csv-config.yaml" -) - -var ErrNoCSVVersion = errors.New("no CSV version supplied") - -type CSV struct { - input.Input - - // ConfigFilePath is the location of a configuration file path for this - // projects' CSV file. - ConfigFilePath string - // CSVVersion is the CSV current version. - CSVVersion string - // FromVersion is the CSV version from which to build a new CSV. A CSV - // manifest with this version should exist at: - // deploy/olm-catalog/{from_version}/operator-name.v{from_version}.{CSVYamlFileExt} - FromVersion string - // OperatorName is the operator's name, ex. app-operator - OperatorName string - - once sync.Once - fs afero.Fs // For testing, ex. afero.NewMemMapFs() - pathPrefix string // For testing, ex. testdata/deploy/olm-catalog -} - -func (s *CSV) initFS(fs afero.Fs) { - s.once.Do(func() { - s.fs = fs - }) -} - -func (s *CSV) getFS() afero.Fs { - s.initFS(afero.NewOsFs()) - return s.fs -} - -func (s *CSV) GetInput() (input.Input, error) { - // A CSV version is required. - if s.CSVVersion == "" { - return input.Input{}, ErrNoCSVVersion - } - if s.Path == "" { - operatorName := strings.ToLower(s.OperatorName) - // Path is what the operator-registry expects: - // {manifests -> olm-catalog}/{operator_name}/{semver}/{operator_name}.v{semver}.clusterserviceversion.yaml - s.Path = filepath.Join(s.pathPrefix, - OLMCatalogDir, - operatorName, - s.CSVVersion, - getCSVFileName(operatorName, s.CSVVersion), - ) - } - if s.ConfigFilePath == "" { - s.ConfigFilePath = filepath.Join(s.pathPrefix, OLMCatalogDir, CSVConfigYamlFile) - } - return s.Input, nil -} - -func (s *CSV) SetFS(fs afero.Fs) { s.initFS(fs) } - -// CustomRender allows a CSV to be written by marshalling -// olmapiv1alpha1.ClusterServiceVersion instead of writing to a template. -func (s *CSV) CustomRender() ([]byte, error) { - s.initFS(afero.NewOsFs()) - - // Get current CSV to update. - csv, exists, err := s.getBaseCSVIfExists() - if err != nil { - return nil, err - } - if !exists { - csv = &olmapiv1alpha1.ClusterServiceVersion{} - } - - cfg, err := GetCSVConfig(s.ConfigFilePath) - if err != nil { - return nil, err - } - - if err = s.updateCSVVersions(csv); err != nil { - return nil, err - } - if err = s.updateCSVFromManifests(cfg, csv); err != nil { - return nil, err - } - s.setCSVDefaultFields(csv) - - if fields := getEmptyRequiredCSVFields(csv); len(fields) != 0 { - if exists { - log.Warnf("Required csv fields not filled in file %s:%s\n", s.Path, joinFields(fields)) - } else { - // A new csv won't have several required fields populated. - // Report required fields to user informationally. - log.Infof("Fill in the following required fields in file %s:%s\n", s.Path, joinFields(fields)) - } - } - - return k8sutil.GetObjectBytes(csv, yaml.Marshal) -} - -func (s *CSV) getBaseCSVIfExists() (*olmapiv1alpha1.ClusterServiceVersion, bool, error) { - verToGet := s.CSVVersion - if s.FromVersion != "" { - verToGet = s.FromVersion - } - csv, exists, err := getCSVFromFSIfExists(s.getFS(), s.getCSVPath(verToGet)) - if err != nil { - return nil, false, err - } - if !exists && s.FromVersion != "" { - log.Warnf("FromVersion set (%s) but CSV does not exist", s.FromVersion) - } - return csv, exists, nil -} - -func getCSVFromFSIfExists(fs afero.Fs, path string) (*olmapiv1alpha1.ClusterServiceVersion, bool, error) { - csvBytes, err := afero.ReadFile(fs, path) - if err != nil { - if os.IsNotExist(err) { - return nil, false, nil - } - return nil, false, err - } - if len(csvBytes) == 0 { - return nil, false, nil - } - - csv := &olmapiv1alpha1.ClusterServiceVersion{} - if err := yaml.Unmarshal(csvBytes, csv); err != nil { - return nil, false, fmt.Errorf("error unmarshalling CSV %s: %v", path, err) - } - - return csv, true, nil -} - -func getCSVName(name, version string) string { - return name + ".v" + version -} - -func getCSVFileName(name, version string) string { - return getCSVName(name, version) + CSVYamlFileExt -} - -func (s *CSV) getCSVPath(ver string) string { - lowerOperatorName := strings.ToLower(s.OperatorName) - name := getCSVFileName(lowerOperatorName, ver) - return filepath.Join(s.pathPrefix, OLMCatalogDir, lowerOperatorName, ver, name) -} - -// setCSVDefaultFields sets all csv fields that should be populated by a user -// to sane defaults. -func (s *CSV) setCSVDefaultFields(csv *olmapiv1alpha1.ClusterServiceVersion) { - // These fields have well-defined required values. - csv.TypeMeta.APIVersion = olmapiv1alpha1.ClusterServiceVersionAPIVersion - csv.TypeMeta.Kind = olmapiv1alpha1.ClusterServiceVersionKind - csv.SetName(getCSVName(strings.ToLower(s.OperatorName), s.CSVVersion)) - - // Set if empty. - if csv.GetNamespace() == "" { - csv.SetNamespace("placeholder") - } - if csv.GetAnnotations() == nil { - csv.SetAnnotations(map[string]string{}) - } - if caps, ok := csv.GetAnnotations()["capabilities"]; !ok || caps == "" { - csv.GetAnnotations()["capabilities"] = "Basic Install" - } - if csv.Spec.Provider == (olmapiv1alpha1.AppLink{}) { - csv.Spec.Provider = olmapiv1alpha1.AppLink{} - } - if len(csv.Spec.Maintainers) == 0 { - csv.Spec.Maintainers = []olmapiv1alpha1.Maintainer{} - } - if len(csv.Spec.Links) == 0 { - csv.Spec.Links = []olmapiv1alpha1.AppLink{} - } - if csv.Spec.DisplayName == "" { - csv.Spec.DisplayName = k8sutil.GetDisplayName(s.OperatorName) - } - if csv.Spec.Description == "" { - csv.Spec.Description = "Placeholder description" - } - if csv.Spec.Maturity == "" { - csv.Spec.Maturity = "alpha" - } - if len(csv.Spec.InstallModes) == 0 { - csv.Spec.InstallModes = []olmapiv1alpha1.InstallMode{ - {Type: olmapiv1alpha1.InstallModeTypeOwnNamespace, Supported: true}, - {Type: olmapiv1alpha1.InstallModeTypeSingleNamespace, Supported: true}, - {Type: olmapiv1alpha1.InstallModeTypeMultiNamespace, Supported: false}, - {Type: olmapiv1alpha1.InstallModeTypeAllNamespaces, Supported: true}, - } - } - if len(csv.Spec.Icon) == 0 { - csv.Spec.Icon = make([]olmapiv1alpha1.Icon, 1) - } - if len(csv.Spec.Keywords) == 0 { - csv.Spec.Keywords = []string{""} - } - if len(csv.Spec.Maintainers) == 0 { - csv.Spec.Maintainers = make([]olmapiv1alpha1.Maintainer, 1) - } - -} - -// TODO: validate that all fields from files are populated as expected -// ex. add `resources` to a CRD - -func getEmptyRequiredCSVFields(csv *olmapiv1alpha1.ClusterServiceVersion) (fields []string) { - // Metadata - if csv.TypeMeta.APIVersion != olmapiv1alpha1.ClusterServiceVersionAPIVersion { - fields = append(fields, "apiVersion") - } - if csv.TypeMeta.Kind != olmapiv1alpha1.ClusterServiceVersionKind { - fields = append(fields, "kind") - } - if csv.ObjectMeta.Name == "" { - fields = append(fields, "metadata.name") - } - // Spec fields - if csv.Spec.Version.String() == "" { - fields = append(fields, "spec.version") - } - if csv.Spec.DisplayName == "" { - fields = append(fields, "spec.displayName") - } - if csv.Spec.Description == "" { - fields = append(fields, "spec.description") - } - if len(csv.Spec.Keywords) == 0 || len(csv.Spec.Keywords[0]) == 0 { - fields = append(fields, "spec.keywords") - } - if len(csv.Spec.Maintainers) == 0 { - fields = append(fields, "spec.maintainers") - } - if csv.Spec.Provider == (olmapiv1alpha1.AppLink{}) { - fields = append(fields, "spec.provider") - } - if csv.Spec.Maturity == "" { - fields = append(fields, "spec.maturity") - } - - return fields -} - -func joinFields(fields []string) string { - sb := &strings.Builder{} - for _, f := range fields { - sb.WriteString("\n\t" + f) - } - return sb.String() -} - -// updateCSVVersions updates csv's version and data involving the version, -// ex. ObjectMeta.Name, and place the old version in the `replaces` object, -// if there is an old version to replace. -func (s *CSV) updateCSVVersions(csv *olmapiv1alpha1.ClusterServiceVersion) error { - - // Old csv version to replace, and updated csv version. - oldVer, newVer := csv.Spec.Version.String(), s.CSVVersion - if oldVer == newVer { - return nil - } - - // Replace all references to the old operator name. - lowerOperatorName := strings.ToLower(s.OperatorName) - oldCSVName := getCSVName(lowerOperatorName, oldVer) - oldRe, err := regexp.Compile(fmt.Sprintf("\\b%s\\b", regexp.QuoteMeta(oldCSVName))) - if err != nil { - return fmt.Errorf("error compiling CSV name regexp %s: %v", oldRe.String(), err) - } - b, err := yaml.Marshal(csv) - if err != nil { - return err - } - newCSVName := getCSVName(lowerOperatorName, newVer) - b = oldRe.ReplaceAll(b, []byte(newCSVName)) - *csv = olmapiv1alpha1.ClusterServiceVersion{} - if err = yaml.Unmarshal(b, csv); err != nil { - return fmt.Errorf("error unmarshalling CSV %s after replacing old CSV name: %v", csv.GetName(), err) - } - - ver, err := semver.Parse(s.CSVVersion) - if err != nil { - return err - } - csv.Spec.Version = olmversion.OperatorVersion{Version: ver} - csv.Spec.Replaces = oldCSVName - return nil -} - -// updateCSVFromManifestFiles gathers relevant data from generated and -// user-defined manifests and updates csv. -func (s *CSV) updateCSVFromManifests(cfg *CSVConfig, csv *olmapiv1alpha1.ClusterServiceVersion) (err error) { - paths := append(cfg.CRDCRPaths, cfg.OperatorPath) - paths = append(paths, cfg.RolePaths...) - manifestGVKMap := map[schema.GroupVersionKind][][]byte{} - crGVKSet := map[schema.GroupVersionKind]struct{}{} - for _, path := range paths { - info, err := s.getFS().Stat(path) - if err != nil { - return err - } - if info.IsDir() { - continue - } - b, err := ioutil.ReadFile(path) - if err != nil { - return err - } - scanner := yamlutil.NewYAMLScanner(b) - for scanner.Scan() { - manifest := scanner.Bytes() - typeMeta, err := k8sutil.GetTypeMetaFromBytes(manifest) - if err != nil { - log.Infof("No TypeMeta in %s, skipping file", path) - continue - } - gvk := typeMeta.GroupVersionKind() - manifestGVKMap[gvk] = append(manifestGVKMap[gvk], manifest) - switch typeMeta.Kind { - case "CustomResourceDefinition": - // Collect CRD kinds to filter them out from unsupported manifest types. - // The CRD type version doesn't matter as long as it has a group, kind, - // and versions in the expected fields. - crd := apiextv1beta1.CustomResourceDefinition{} - if err = yaml.Unmarshal(manifest, &crd); err != nil { - return err - } - for _, ver := range crd.Spec.Versions { - crGVK := schema.GroupVersionKind{ - Group: crd.Spec.Group, - Version: ver.Name, - Kind: crd.Spec.Names.Kind, - } - crGVKSet[crGVK] = struct{}{} - } - } - } - if err = scanner.Err(); err != nil { - return err - } - } - - crUpdaters := crs{} - for gvk, manifests := range manifestGVKMap { - // We don't necessarily care about sorting by a field value, more about - // consistent ordering. - sort.Slice(manifests, func(i int, j int) bool { - return string(manifests[i]) < string(manifests[j]) - }) - switch gvk.Kind { - case "Role": - err = roles(manifests).apply(csv) - case "ClusterRole": - err = clusterRoles(manifests).apply(csv) - case "Deployment": - err = deployments(manifests).apply(csv) - case "CustomResourceDefinition": - // TODO(estroz): customresourcedefinition should not be updated for - // Ansible and Helm CSV's until annotated updates are implemented. - if projutil.IsOperatorGo() { - err = crds(manifests).apply(csv) - } - default: - if _, ok := crGVKSet[gvk]; ok { - crUpdaters = append(crUpdaters, manifests...) - } else { - log.Infof("Skipping manifest %s", gvk) - } - } - if err != nil { - return err - } - } - // Re-sort CR's since they are appended in random order. - sort.Slice(crUpdaters, func(i int, j int) bool { - return string(crUpdaters[i]) < string(crUpdaters[j]) - }) - if err = crUpdaters.apply(csv); err != nil { - return err - } - return nil -} diff --git a/internal/scaffold/olm-catalog/csv_test.go b/internal/scaffold/olm-catalog/csv_test.go deleted file mode 100644 index 46a00118a70..00000000000 --- a/internal/scaffold/olm-catalog/csv_test.go +++ /dev/null @@ -1,207 +0,0 @@ -// Copyright 2018 The Operator-SDK Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package catalog - -import ( - "bytes" - "io" - "io/ioutil" - "os" - "path/filepath" - "testing" - - "github.com/operator-framework/operator-sdk/internal/scaffold" - "github.com/operator-framework/operator-sdk/internal/scaffold/input" - testutil "github.com/operator-framework/operator-sdk/internal/scaffold/internal/testutil" - "github.com/operator-framework/operator-sdk/internal/util/diffutil" - "github.com/operator-framework/operator-sdk/pkg/k8sutil" - - "github.com/blang/semver" - "github.com/ghodss/yaml" - olmapiv1alpha1 "github.com/operator-framework/operator-lifecycle-manager/pkg/api/apis/operators/v1alpha1" - "github.com/spf13/afero" - appsv1 "k8s.io/api/apps/v1" -) - -const ( - testDataDir = "testdata" - projectName = "app-operator-dir" - operatorName = "app-operator" - oldCSVVer = "0.1.0" - newCSVVer = "0.2.0" - csvVer = "0.1.0" -) - -var testDeployDir = filepath.Join(testDataDir, scaffold.DeployDir) - -func TestCSVNew(t *testing.T) { - buf := &bytes.Buffer{} - s := &scaffold.Scaffold{ - GetWriter: func(_ string, _ os.FileMode) (io.Writer, error) { - return buf, nil - }, - } - - sc := &CSV{CSVVersion: csvVer, pathPrefix: testDataDir, OperatorName: operatorName} - err := s.Execute(&input.Config{ProjectName: projectName}, sc) - if err != nil { - t.Fatalf("Failed to execute the scaffold: (%v)", err) - } - - // Get the expected CSV manifest from test data dir. - csvExpBytes, err := afero.ReadFile(s.Fs, sc.getCSVPath(csvVer)) - if err != nil { - t.Fatal(err) - } - csvExp := string(csvExpBytes) - if csvExp != buf.String() { - diffs := diffutil.Diff(csvExp, buf.String()) - t.Errorf("Expected vs actual differs.\n%v", diffs) - } -} - -func TestCSVFromOld(t *testing.T) { - s := &scaffold.Scaffold{Fs: afero.NewMemMapFs()} - - // Write all files in testdata/deploy to fs so manifests are present when - // writing a new CSV. - if err := testutil.WriteOSPathToFS(afero.NewOsFs(), s.Fs, testDeployDir); err != nil { - t.Fatalf("Failed to write %s to in-memory test fs: (%v)", testDeployDir, err) - } - - sc := &CSV{ - CSVVersion: newCSVVer, - FromVersion: oldCSVVer, - pathPrefix: testDataDir, - OperatorName: operatorName, - } - err := s.Execute(&input.Config{ProjectName: projectName}, sc) - if err != nil { - t.Fatalf("Failed to execute the scaffold: (%v)", err) - } - - // Check if a new file was written at the expected path. - newCSVPath := sc.getCSVPath(newCSVVer) - newCSV, newExists, err := getCSVFromFSIfExists(s.Fs, newCSVPath) - if err != nil { - t.Fatalf("Failed to get new CSV %s: (%v)", newCSVPath, err) - } - if !newExists { - t.Fatalf("New CSV does not exist at %s", newCSVPath) - } - - expName := getCSVName(operatorName, newCSVVer) - if newCSV.ObjectMeta.Name != expName { - t.Errorf("Expected CSV metadata.name %s, got %s", expName, newCSV.ObjectMeta.Name) - } - expReplaces := getCSVName(operatorName, oldCSVVer) - if newCSV.Spec.Replaces != expReplaces { - t.Errorf("Expected CSV spec.replaces %s, got %s", expReplaces, newCSV.Spec.Replaces) - } -} - -func TestUpdateVersion(t *testing.T) { - sc := &CSV{ - Input: input.Input{ProjectName: projectName}, - CSVVersion: newCSVVer, - pathPrefix: testDataDir, - OperatorName: operatorName, - } - csvExpBytes, err := ioutil.ReadFile(sc.getCSVPath(oldCSVVer)) - if err != nil { - t.Fatal(err) - } - csv := &olmapiv1alpha1.ClusterServiceVersion{} - if err := yaml.Unmarshal(csvExpBytes, csv); err != nil { - t.Fatal(err) - } - - if err := sc.updateCSVVersions(csv); err != nil { - t.Fatalf("Failed to update csv with version %s: (%v)", newCSVVer, err) - } - - wantedSemver, err := semver.Parse(newCSVVer) - if err != nil { - t.Errorf("Failed to parse %s: %v", newCSVVer, err) - } - if !csv.Spec.Version.Equals(wantedSemver) { - t.Errorf("Wanted csv version %v, got %v", wantedSemver, csv.Spec.Version) - } - wantedName := getCSVName(operatorName, newCSVVer) - if csv.ObjectMeta.Name != wantedName { - t.Errorf("Wanted csv name %s, got %s", wantedName, csv.ObjectMeta.Name) - } - - csvDepSpecs := csv.Spec.InstallStrategy.StrategySpec.DeploymentSpecs - if len(csvDepSpecs) != 1 { - t.Fatal("No deployment specs in CSV") - } - csvPodImage := csvDepSpecs[0].Spec.Template.Spec.Containers[0].Image - if len(csvDepSpecs[0].Spec.Template.Spec.Containers) != 1 { - t.Fatal("No containers in CSV deployment spec") - } - // updateCSVVersions should not update podspec image. - wantedImage := "quay.io/example-inc/operator:v0.1.0" - if csvPodImage != wantedImage { - t.Errorf("Podspec image changed from %s to %s", wantedImage, csvPodImage) - } - - wantedReplaces := getCSVName(operatorName, "0.1.0") - if csv.Spec.Replaces != wantedReplaces { - t.Errorf("Wanted csv replaces %s, got %s", wantedReplaces, csv.Spec.Replaces) - } -} - -func TestSetAndCheckOLMNamespaces(t *testing.T) { - depBytes, err := ioutil.ReadFile(filepath.Join(testDeployDir, "operator.yaml")) - if err != nil { - t.Fatalf("Failed to read Deployment bytes: %v", err) - } - - // The test operator.yaml doesn't have "olm.targetNamespaces", so first - // check that depHasOLMNamespaces() returns false. - dep := appsv1.Deployment{} - if err := yaml.Unmarshal(depBytes, &dep); err != nil { - t.Fatalf("Failed to unmarshal Deployment bytes: %v", err) - } - if depHasOLMNamespaces(dep) { - t.Error("Expected depHasOLMNamespaces to return false, got true") - } - - // Insert "olm.targetNamespaces" into WATCH_NAMESPACE and check that - // depHasOLMNamespaces() returns true. - setWatchNamespacesEnv(&dep) - if !depHasOLMNamespaces(dep) { - t.Error("Expected depHasOLMNamespaces to return true, got false") - } - - // Overwrite WATCH_NAMESPACE and check that depHasOLMNamespaces() returns - // false. - overwriteContainerEnvVar(&dep, k8sutil.WatchNamespaceEnvVar, newEnvVar("FOO", "bar")) - if depHasOLMNamespaces(dep) { - t.Error("Expected depHasOLMNamespaces to return false, got true") - } - - // Insert "olm.targetNamespaces" elsewhere in the deployment pod spec - // and check that depHasOLMNamespaces() returns true. - dep = appsv1.Deployment{} - if err := yaml.Unmarshal(depBytes, &dep); err != nil { - t.Fatalf("Failed to unmarshal Deployment bytes: %v", err) - } - dep.Spec.Template.ObjectMeta.Labels["namespace"] = olmTNMeta - if !depHasOLMNamespaces(dep) { - t.Error("Expected depHasOLMNamespaces to return true, got false") - } -} diff --git a/internal/util/k8sutil/crd.go b/internal/util/k8sutil/crd.go index 32a521fbca0..2d974a0edc2 100644 --- a/internal/util/k8sutil/crd.go +++ b/internal/util/k8sutil/crd.go @@ -23,6 +23,7 @@ import ( "regexp" yaml "github.com/ghodss/yaml" + log "github.com/sirupsen/logrus" apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" "k8s.io/apimachinery/pkg/version" ) @@ -48,28 +49,34 @@ func GetCRDs(crdsDir string) ([]*apiextv1beta1.CustomResourceDefinition, error) return crds, nil } -// GetCRDManifestPaths gets all CRD manifest paths in crdsDir and subdirs. +// GetCRDManifestPaths returns all CRD manifest paths in crdsDir and subdirs. func GetCRDManifestPaths(crdsDir string) (crdPaths []string, err error) { err = filepath.Walk(crdsDir, func(path string, info os.FileInfo, werr error) error { if werr != nil { return werr } - if info == nil { + + // Only read manifest from files, not directories + if info.IsDir() { return nil } - if !info.IsDir() { - b, err := ioutil.ReadFile(path) - if err != nil { - return fmt.Errorf("error reading manifest %s: %v", path, err) - } - typeMeta, err := GetTypeMetaFromBytes(b) - if err != nil { - return fmt.Errorf("error getting kind from manifest %s: %v", path, err) - } - if typeMeta.Kind == "CustomResourceDefinition" { - crdPaths = append(crdPaths, path) - } + + b, err := ioutil.ReadFile(path) + if err != nil { + return fmt.Errorf("error reading manifest %s: %v", path, err) + } + // Skip files in crdsDir that aren't k8s manifests since we do not know + // what other files are in crdsDir. + typeMeta, err := GetTypeMetaFromBytes(b) + if err != nil { + log.Debugf("Skipping non-manifest file %s: %v", path, err) + return nil + } + if typeMeta.Kind != "CustomResourceDefinition" { + log.Debugf("Skipping non CRD manifest %s", path) + return nil } + crdPaths = append(crdPaths, path) return nil }) return crdPaths, err diff --git a/internal/util/projutil/project_util.go b/internal/util/projutil/project_util.go index 3b47d0dbc94..4ab7c609a12 100644 --- a/internal/util/projutil/project_util.go +++ b/internal/util/projutil/project_util.go @@ -77,6 +77,8 @@ func MustInProjectRoot() { // CheckProjectRoot checks if the current dir is the project root, and returns // an error if not. +// TODO(hasbro17): Change this to check for go.mod +// "build/Dockerfile" may not be present in all projects func CheckProjectRoot() error { // If the current directory has a "build/Dockerfile", then it is safe to say // we are at the project root. @@ -113,9 +115,19 @@ func getHomeDir() (string, error) { return homedir.Expand(hd) } +// TODO(hasbro17): If this function is called in the subdir of +// a module project it will fail to parse go.mod and return +// the correct import path. +// This needs to be fixed to return the pkg import path for any subdir +// in order for `generate csv` to correctly form pkg imports +// for API pkg paths that are not relative to the root dir. +// This might not be fixable since there is no good way to +// get the project root from inside the subdir of a module project. +// // GetGoPkg returns the current directory's import path by parsing it from // wd if this project's repository path is rooted under $GOPATH/src, or // from go.mod the project uses Go modules to manage dependencies. +// If the project has a go.mod then wd must be the project root. // // Example: "github.com/example-inc/app-operator" func GetGoPkg() string { diff --git a/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.2/memcached-operator.v0.0.2.clusterserviceversion.yaml b/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.2/memcached-operator.v0.0.2.clusterserviceversion.yaml index d2c48a2ddf3..663ab2647d4 100644 --- a/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.2/memcached-operator.v0.0.2.clusterserviceversion.yaml +++ b/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.2/memcached-operator.v0.0.2.clusterserviceversion.yaml @@ -1,30 +1,112 @@ -# This file defines the ClusterServiceVersion (CSV) to tell the catalog how to display, create and -# manage the application as a whole. If changes are made to the CRD for this application kind, -# make sure to replace those references below as well. apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: - alm-examples: '[{"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"name":"example-memcached"},"spec":{"size":3}}]' + alm-examples: |- + [ + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "Memcached", + "metadata": { + "name": "example-memcached" + }, + "spec": { + "size": 3 + } + }, + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "MemcachedRS", + "metadata": { + "name": "example-memcachedrs" + }, + "spec": { + "numNodes": 4 + } + } + ] capabilities: Basic Install name: memcached-operator.v0.0.2 namespace: placeholder spec: - installModes: - - type: OwnNamespace - supported: true - - type: SingleNamespace - supported: true - - type: MultiNamespace - supported: false - - type: AllNamespaces - supported: true + apiservicedefinitions: {} + customresourcedefinitions: + owned: + - kind: MemcachedRS + name: memcachedrs.cache.example.com + version: v1alpha1 + - description: Represents a cluster of Memcached apps + displayName: Memcached App + kind: Memcached + name: memcacheds.cache.example.com + resources: + - kind: Deployment + name: "" + version: v1 + - kind: ReplicaSet + name: "" + version: v1 + - kind: Pod + name: "" + version: v1 + specDescriptors: + - description: The desired number of member Pods for the deployment. + displayName: Size + path: size + x-descriptors: + - urn:alm:descriptor:com.tectonic.ui:podCount + statusDescriptors: + - description: The current status of the application. + displayName: Status + path: phase + x-descriptors: + - urn:alm:descriptor:io.kubernetes.phase + - description: Explanation for the current status of the application. + displayName: Status Details + path: reason + x-descriptors: + - urn:alm:descriptor:io.kubernetes.phase:reason + version: v1alpha1 + - kind: NotExistKind + name: notexist.example.com + version: v1alpha1 + description: Main enterprise application providing business critical features with + high availability and no manual intervention. + displayName: Memcached Application install: - strategy: deployment spec: + deployments: + - name: memcached-operator + spec: + replicas: 1 + selector: + matchLabels: + name: memcached-operator + template: + metadata: + labels: + name: memcached-operator + spec: + containers: + - command: + - memcached-operator + env: + - name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.annotations['olm.targetNamespaces'] + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPERATOR_NAME + value: memcached-operator + image: quay.io/coreos/operator-sdk-dev:test-framework-operator + imagePullPolicy: Always + name: memcached-operator + serviceAccountName: memcached-operator permissions: - - serviceAccountName: memcached-operator - rules: + - rules: - apiGroups: - "" resources: @@ -55,83 +137,31 @@ spec: - '*' - apiGroups: - apps - resources: - - deployments/finalizers resourceNames: - memcached-operator + resources: + - deployments/finalizers verbs: - - "update" - deployments: - - name: memcached-operator - spec: - replicas: 1 - selector: - matchLabels: - name: memcached-operator - template: - metadata: - labels: - name: memcached-operator - spec: - serviceAccountName: memcached-operator - containers: - - name: memcached-operator - image: quay.io/coreos/operator-sdk-dev:test-framework-operator - command: - - memcached-operator - imagePullPolicy: Always - env: - - name: WATCH_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.annotations['olm.targetNamespaces'] - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: OPERATOR_NAME - value: "memcached-operator" - customresourcedefinitions: - owned: - - description: Represents a cluster of Memcached apps - displayName: Memcached App - kind: Memcached - name: memcacheds.cache.example.com - version: v1alpha1 - resources: - - kind: Deployment - version: v1 - - kind: ReplicaSet - version: v1beta2 - - kind: Pod - version: v1 - specDescriptors: - - description: The desired number of member Pods for the deployment. - displayName: Size - path: size - x-descriptors: - - 'urn:alm:descriptor:com.tectonic.ui:podCount' - statusDescriptors: - - description: The current status of the application. - displayName: Status - path: phase - x-descriptors: - - 'urn:alm:descriptor:io.kubernetes.phase' - - description: Explanation for the current status of the application. - displayName: Status Details - path: reason - x-descriptors: - - 'urn:alm:descriptor:io.kubernetes.phase:reason' + - update + serviceAccountName: memcached-operator + strategy: deployment + installModes: + - supported: true + type: OwnNamespace + - supported: true + type: SingleNamespace + - supported: false + type: MultiNamespace + - supported: true + type: AllNamespaces keywords: - memcached - app - displayName: Memcached Application + maintainers: + - email: corp@example.com + name: Some Corp + maturity: alpha provider: name: Example url: www.example.com - maturity: alpha version: 0.0.2 - maintainers: - - email: corp@example.com - name: Some Corp - description: Main enterprise application providing business critical features with high availability and no manual intervention. diff --git a/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml b/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml index c331fa07de4..e2e61f4a81b 100644 --- a/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml +++ b/test/test-framework/deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml @@ -13,6 +13,16 @@ metadata: "spec": { "size": 3 } + }, + { + "apiVersion": "cache.example.com/v1alpha1", + "kind": "MemcachedRS", + "metadata": { + "name": "example-memcachedrs" + }, + "spec": { + "numNodes": 4 + } } ] capabilities: Basic Install @@ -22,6 +32,18 @@ spec: apiservicedefinitions: {} customresourcedefinitions: owned: + - description: Represents a cluster of MemcachedRS apps + displayName: MemcachedRS App + kind: MemcachedRS + name: memcachedrs.cache.example.com + statusDescriptors: + - description: List of the pod names running Memcached in the cluster + displayName: Nodes + path: nodeList + - description: A useless testing variable + displayName: Test + path: test + version: v1alpha1 - description: Represents a cluster of Memcached apps displayName: Memcached App kind: Memcached @@ -54,18 +76,6 @@ spec: x-descriptors: - urn:alm:descriptor:io.kubernetes.phase:reason version: v1alpha1 - - description: Represents a cluster of MemcachedRS apps - displayName: MemcachedRS App - kind: MemcachedRS - name: memcachedrs.cache.example.com - version: v1alpha1 - statusDescriptors: - - description: List of the pod names running Memcached in the cluster - displayName: Nodes - path: nodeList - - description: A useless testing variable - displayName: Test - path: test description: Main enterprise application providing business critical features with high availability and no manual intervention. displayName: Memcached Application diff --git a/website/content/en/docs/cli/operator-sdk_generate_csv.md b/website/content/en/docs/cli/operator-sdk_generate_csv.md index 1b20934ca18..57d80b6d193 100644 --- a/website/content/en/docs/cli/operator-sdk_generate_csv.md +++ b/website/content/en/docs/cli/operator-sdk_generate_csv.md @@ -11,22 +11,109 @@ A CSV semantic version is supplied via the --csv-version flag. If your operator has already generated a CSV manifest you want to use as a base, supply its version to --from-version. Otherwise the SDK will scaffold a new CSV manifest. -Configure CSV generation by writing a config file 'deploy/olm-catalog/csv-config.yaml +CSV input flags: + --deploy-dir: + The CSV's install strategy and permissions will be generated from the operator manifests + (Deployment and Role/ClusterRole) present in this directory. + + --apis-dir: + The CSV annotation comments will be parsed from the Go types under this path to + fill out metadata for owned APIs in spec.customresourcedefinitions.owned. + + --crd-dir: + The CSV's spec.customresourcedefinitions.owned field is generated from the CRD manifests + in this path.These CRD manifests are also copied over to the bundle directory if --update-crds is set. + Additionally the CR manifests will be used to populate the CSV example CRs. + ``` operator-sdk generate csv [flags] ``` +### Examples + +``` + + ##### Generate CSV from default input paths ##### + $ tree pkg/apis/ deploy/ + pkg/apis/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + deploy/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree deploy/ + deploy/ + ... + ├── olm-catalog + │   └── memcached-operator + │   ├── 0.0.1 + │   │   ├── cache.example.com_memcacheds_crd.yaml + │   │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + │   └── memcached-operator.package.yaml + ... + + + + ##### Generate CSV from custom input paths ##### + $ operator-sdk generate csv --csv-version=0.0.1 --update-crds \ + --deploy-dir=config --apis-dir=api --output-dir=production + INFO[0000] Generating CSV manifest version 0.0.1 + ... + + $ tree config/ api/ production/ + config/ + ├── crds + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── cache.example.com_v1alpha1_memcached_cr.yaml + ├── operator.yaml + ├── role.yaml + ├── role_binding.yaml + └── service_account.yaml + api/ + ├── ... + └── cache + ├── group.go + └── v1alpha1 + ├── ... + └── memcached_types.go + production/ + └── olm-catalog + └── memcached-operator + ├── 0.0.1 + │   ├── cache.example.com_memcacheds_crd.yaml + │   └── memcached-operator.v0.0.1.clusterserviceversion.yaml + └── memcached-operator.package.yaml + +``` + ### Options ``` + --apis-dir string Project relative path to root directory for API type defintions (default "pkg/apis") + --crd-dir string Project relative path to root directory for CRD and CR manifests (default "deploy/crds") --csv-channel string Channel the CSV should be registered under in the package manifest - --csv-config string Path to CSV config file. Defaults to deploy/olm-catalog/csv-config.yaml --csv-version string Semantic version of the CSV --default-channel Use the channel passed to --csv-channel as the package manifests' default channel. Only valid when --csv-channel is set + --deploy-dir string Project relative path to root directory for operator manifests (Deployment and RBAC) (default "deploy") --from-version string Semantic version of an existing CSV to use as a base -h, --help help for csv --operator-name string Operator name to use while generating CSV + --output-dir string Base directory to output generated CSV. The resulting CSV bundle directory will be "/olm-catalog//". (default "deploy") --update-crds Update CRD manifests in deploy/{operator-name}/{csv-version} the using latest API's ``` diff --git a/website/content/en/docs/golang/olm-catalog/generating-a-csv.md b/website/content/en/docs/golang/olm-catalog/generating-a-csv.md index 4db5492e0d8..3f6782a25a1 100644 --- a/website/content/en/docs/golang/olm-catalog/generating-a-csv.md +++ b/website/content/en/docs/golang/olm-catalog/generating-a-csv.md @@ -10,35 +10,23 @@ This document describes how to manage the following lifecycle for your Operator ## Configuration -Operator SDK projects have an expected [project layout][doc-project-layout]. In particular, a few manifests are expected to be present in the `deploy` directory: +### Inputs -* Roles: `role.yaml` -* Deployments: `operator.yaml` -* Custom Resources (CR's): `crds/___cr.yaml` -* Custom Resource Definitions (CRD's): `crds/__crd.yaml`. +The CSV generator requires certain inputs to construct a CSV manifest. -`generate csv` reads these files and adds their data to a CSV in an alternate form. +1. Path to the operator manifests root directory. By default `generate csv` extracts manifests from files in `deploy/` for the following kinds and adds them to the CSV. Use the `--deploy-dir` flag to change this path. + * Roles: `role.yaml` + * ClusterRoles: `cluster_role.yaml` + * Deployments: `operator.yaml` + * Custom Resources (CR's): `crds/___cr.yaml` + * CustomResourceDefinitions (CRD's): `crds/__crd.yaml` +2. Path to API types root directory. The CSV generator also parses the [CSV annotations][csv-annotations] from the API type definitions to populate certain CSV fields. By default the API types directory is `pkg/apis/`. Use the `--apis-dir` flag to change this path. The CSV generator expects either of the following layouyts for the API types directory + * Mulitple groups: `///` + * Single groups: `//` -The following example config containing default values should be copied and written to `deploy/olm-catalog/csv-config.yaml`: +### Output -```yaml -crd-cr-paths: -- deploy/crds -operator-path: deploy/operator.yaml -role-paths: -- deploy/role.yaml -``` - -Explanation of all config fields: - -- `crd-cr-paths`: list of strings - a list of CRD and CR manifest file/directory paths. Defaults to `[deploy/crds]`. -- `operator-path`: string - the operator `Deployment` manifest file path. Defaults to `deploy/operator.yaml`. -- `role-paths`: list of strings - Role and ClusterRole manifest file paths. Defaults to `[deploy/role.yaml]`. -- `operator-name`: string - the name used to create the CSV and manifest file names. Defaults to the project's name. - -**Note**: The [design doc][doc-csv-design] has outdated field information which should not be referenced. - -Fields in this config file can be modified to point towards alternate manifest locations, and passed to `generate csv --csv-config=` to configure CSV generation. For example, if I have one set of production CR/CRD manifests under `deploy/crds/production`, and a set of test manifests under `deploy/crds/test`, and I only want to include production manifests in my CSV, I can set `crd-cr-paths: [deploy/crds/production]`. `generate csv` will then ignore `deploy/crds/test` when getting CR/CRD data. +By default `generate csv` will generate the catalog bundle directory `olm-catalog/...` under `deploy/`. To change where the CSV bundle directory is generated use the `--ouput-dir` flag. ## Versioning @@ -46,7 +34,7 @@ CSV's are versioned in path, file name, and in their `metadata.name` field. For `generate csv` allows you to upgrade your CSV using the `--from-version` flag. If you have an existing CSV with version `0.0.1` and want to write a new version `0.0.2`, you can run `operator-sdk generate csv --csv-version 0.0.2 --from-version 0.0.1`. This will write a new CSV manifest to `deploy/olm-catalog//0.0.2/.v0.0.2.clusterserviceversion.yaml` containing user-defined data from `0.0.1` and any modifications you've made to `roles.yaml`, `operator.yaml`, CR's, or CRD's. -The SDK can manage CRD's in your Operator bundle as well. You can pass the `--update-crds` flag to `generate csv` to add or update your CRD's in your bundle by copying manifests pointed to by `crd-cr-paths` in your config. CRD's in a bundle are not updated by default. +The SDK can manage CRD's in your Operator bundle as well. You can pass the `--update-crds` flag to `generate csv` to add or update your CRD's in your bundle by copying manifests in `deploy/crds` to your bundle. CRD's in a bundle are not updated by default. ## First Generation @@ -86,7 +74,7 @@ Be sure to include the `--update-crds` flag if you want to add CRD's to your bun Below are two lists of fields: the first is a list of all fields the SDK and OLM expect in a CSV, and the second are optional. -Several fields require user input (labeled _user_) or a [code annotation][code-annotations] (labeled _annotation_). This list may change as the SDK becomes better at generating CSV's. +Several fields require user input (labeled _user_) or a [CSV annotation][csv-annotations] (labeled _annotation_). This list may change as the SDK becomes better at generating CSV's. Required: @@ -130,10 +118,9 @@ Optional: [doc-csv]:https://github.com/operator-framework/operator-lifecycle-manager/blob/4197455/Documentation/design/building-your-csv.md [olm]:https://github.com/operator-framework/operator-lifecycle-manager [generate-csv-cli]:../../cli/operator-sdk_generate_csv.md -[doc-project-layout]:../../project_layout.md [doc-csv-design]:../../design/milestone-0.2.0/csv-generation.md [doc-bundle]:https://github.com/operator-framework/operator-registry/blob/6893d19/README.md#manifest-format [x-desc-list]:https://github.com/openshift/console/blob/70bccfe/frontend/public/components/operator-lifecycle-manager/descriptors/types.ts#L3-L35 [install-modes]:https://github.com/operator-framework/operator-lifecycle-manager/blob/4197455/Documentation/design/building-your-csv.md#operator-metadata [olm-capabilities]:../../images/operator-capability-level.png -[code-annotations]:../../proposals/sdk-code-annotations.md +[csv-annotations]: ./csv-annotations.md