diff --git a/docs/content/en/docs/concepts/_index.md b/docs/content/en/docs/concepts/_index.md index a1e9979cd2..050289e4b9 100644 --- a/docs/content/en/docs/concepts/_index.md +++ b/docs/content/en/docs/concepts/_index.md @@ -79,9 +79,3 @@ Currently, PipeCD is supporting these four cloud providers: `KUBERNETES`, `TERRA PipeCD supports multiple methods to automate the analysis process of the deployment. It can be done by using metrics, logs or by checking the configured http requests. Analysis Provider defines where to get those metrics/log data, like `Prometheus`, `Datadog`, `Stackdriver`, `CloudWatch`, and so on. - -### Image Provider - -PipeCD can automatically trigger a new Deployment when a new image tag stored at a container registry is pushed by enabling the [Image watcher](/docs/user-guide/image-watcher) feature. -Image Provider defines which container registry should be monitored. -Currently, PipeCD is supporting only `ECR`. `GCR` and `DOCKERHUB` are on the roadmap. diff --git a/docs/content/en/docs/operator-manual/piped/configuration-reference.md b/docs/content/en/docs/operator-manual/piped/configuration-reference.md index 0a70df2f4a..28e9d47d72 100644 --- a/docs/content/en/docs/operator-manual/piped/configuration-reference.md +++ b/docs/content/en/docs/operator-manual/piped/configuration-reference.md @@ -30,8 +30,6 @@ spec: | chartRepositories | [][ChartRepository](/docs/operator-manual/piped/configuration-reference/#chartrepository) | List of Helm chart repositories that should be added while starting up. | No | | cloudProviders | [][CloudProvider](/docs/operator-manual/piped/configuration-reference/#cloudprovider) | List of cloud providers can be used by this piped. | No | | analysisProviders | [][AnalysisProvider](/docs/operator-manual/piped/configuration-reference/#analysisprovider) | List of analysis providers can be used by this piped. | No | -| imageProviders | [][ImageProvider](/docs/operator-manual/piped/configuration-reference/#imageprovider) | List of image providers can be used by this piped. | No | -| imageWatcher | [ImageWatcher](/docs/operator-manual/piped/configuration-reference/#imagewatcher) | Optional Image watcher settings for each git repository | No | | eventWatcher | [EventWatcher](/docs/operator-manual/piped/configuration-reference/#eventwatcher) | Optional Event watcher settings | No | | notifications | [Notifications](/docs/operator-manual/piped/configuration-reference/#notifications) | Sending notifications to Slack, Webhook... | No | @@ -137,49 +135,6 @@ Must be one of the following structs: | usernameFile | string | The path to the username file. | No | | passwordFile | string | The path to the password file. | No | -## ImageProvider - -| Field | Type | Description | Required | -|-|-|-|-| -| name | string | The unique name of the analysis provider. | Yes | -| type | string | The provider type. Currently, `GCR` and `ECR` are available. | Yes | -| config | [ImageProviderConfig](/docs/operator-manual/piped/configuration-reference/#imageproviderconfig) | Specific configuration for the specified type of image provider. | Yes | - -## ImageProviderConfig - -Must be one of the following structs: - -### ImageProviderGCRConfig - -| Field | Type | Description | Required | -|-|-|-|-| -| serviceAccountFile | string | The path to the json file of service account with the required `roles/storage.objectViewer` role. | No | - -### ImageProviderECRConfig - -| Field | Type | Description | Required | -|-|-|-|-| -| region | string | The region to send requests to. This parameter is required. e.g. "us-west-2". A full list of regions is: https://docs.aws.amazon.com/general/latest/gr/rande.html | Yes | -| registryId | string | The AWS account ID associated with the registry that contains the repository in which to list images. The "default" registry is assumed by default. | No | -| credentialsFile | string | Path to the shared credentials file. | No | -| profile | string | AWS Profile to extract credentials from the shared credentials file. If empty, the environment variable "AWS_PROFILE" is used. "default" is populated if the environment variable is also not set. | No | - -## ImageWatcher - -| Field | Type | Description | Required | -|-|-|-|-| -| checkInterval | duration | Interval to compare the image in the git repository and one in the images provider. Defaults to `5m`. | No | -| gitRepos | [][ImageWatcherGitRepo](/docs/operator-manual/piped/configuration-reference/#imagewatchergitrepo) | List of settings for each git repository | No | - -### ImageWatcherGitRepo - -| Field | Type | Description | Required | -|-|-|-|-| -| repoId | string | Id of the git repository. This must be unique within the repos' elements. | Yes | -| commitMessage | string | The commit message used to push after updating image. Default message is used if not given. | No | -| includes | []string | The paths to ImageWatcher files to be included. | No | -| excludes | []string | The paths to ImageWatcher files to be excluded. This is prioritized if both includes and this are given. | No | - ## EventWatcher | Field | Type | Description | Required | diff --git a/docs/content/en/docs/operator-manual/piped/configuring-image-watcher.md b/docs/content/en/docs/operator-manual/piped/configuring-image-watcher.md deleted file mode 100644 index e747a1b556..0000000000 --- a/docs/content/en/docs/operator-manual/piped/configuring-image-watcher.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: "Configuring image watcher" -linkTitle: "Configuring image watcher" -weight: 10 -description: > - This page describes how to configure piped to enable image watcher. ---- - -To enable [ImageWatcher](/docs/user-guide/image-watcher/), you have to configure your piped at first. - -## Prerequisites -The [SSH key Piped use](/docs/operator-manual/piped/configuration-reference/#git) must be a key with write-access because Image watcher automates the deployment flow by commitng and pushing to your git repository. - -## Adding an image provider -Define arbitrary number of [image providers](/docs/concepts#image-provider) which is information needed to connect from your Piped to the container registry. -It will run a pull operation every 5 minutes by default. This interval can be set in the `imageWatcher` field touch upon later. -Also, we plan to provide a FAKE image provider mentioned below to avoid the rate limit. - -Currently, PipeCD is supporting: -- [Google Container Registry (GCR)](https://cloud.google.com/container-registry) -- [Amazon Elastic Container Registry (ECR)](https://aws.amazon.com/ecr) - -### GCR -Append the `GCR` image provider to the Piped configuration file as: - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: Piped -spec: - imageProviders: - - name: my-gcr - type: GCR - config: - serviceAccountFile: /etc/piped-secret/gcr-service-account.json -``` - -For public repositories, no configuration is required. - -If you want to watch private repository, you should set up authentication. -A [service account](https://cloud.google.com/compute/docs/access/service-accounts) is the only authentication way currently available. -You give the path to the json file of service account with the required `roles/storage.objectViewer` role. - -The full list of GCR fields are [here](/docs/operator-manual/piped/configuration-reference/#imageprovidergcrconfig). - -### ECR - ->NOTE: Currently, it supports only ECR private repositories. - -Append the `ECR` image provider to the Piped configuration file as: - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: Piped -spec: - imageProviders: - - name: my-ecr - type: ECR - config: - region: ap-northeast-1 - credentialsFile: /etc/piped-secret/aws-credentials - profile: user1 -``` - -The only required field is `region`. - -You will generally need your AWS credentials to authenticate with ECR. Piped provides multiple methods of loading these credentials. -It attempts to retrieve credentials in the following order: -1. From the environment variables. Available environment variables are `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY` -1. From the given credentials file. -1. From the EC2 Instance Role - -Hence, you don't have to set `credentialsFile` if you use the environment variables or the EC2 Instance Role. Keep in mind the IAM role/user that you use with your Piped must possess the IAM policy permission for `ecr:DescribeImages`. - -The full list of ECR fields are [here](/docs/operator-manual/piped/configuration-reference/#imageproviderecrconfig). - -### DockerHub - ->TBA - -### FAKE - ->TBA: We plan to provide a FAKE image provider to deal with the rate limit from the container registry. -> ->The FAKE container registry is deployed at the control-plane, and you can store the metadata about newly updated images whenever you want to (e.g. on your CI). - - -## [optional] Settings for watcher -The Piped's behavior can be finely controlled by setting the `imageWatcher` field. - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: Piped -spec: - imageWatcher: - checkInterval: 5m - gitRepos: - - repoId: foo - commitMessage: Update image - includes: - - imagewatcher-dev.yaml - - imagewatcher-stg.yaml -``` - -If multiple Pipeds handle a single repository, you can prevent conflicts by splitting into the multiple ImageWatcher files and setting `includes/excludes` to specify the files that should be monitored by this Piped. -`excludes` is prioritized if both `includes` and `excludes` are given. - -The full list of configurable fields are [here](/docs/operator-manual/piped/configuration-reference/#imagewatcher). - -## [optional] Settings for git user -By default, every git commit uses `piped` as a username and `pipecd.dev@gmail.com` as an email. You can change it with the [git](/docs/operator-manual/piped/configuration-reference/#git) field. - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: Piped -spec: - git: - username: foo - email: foo@example.com -``` diff --git a/docs/content/en/docs/user-guide/configuration-reference.md b/docs/content/en/docs/user-guide/configuration-reference.md index 7b6f2d45cf..f35cb4cdb6 100644 --- a/docs/content/en/docs/user-guide/configuration-reference.md +++ b/docs/content/en/docs/user-guide/configuration-reference.md @@ -110,32 +110,6 @@ spec: |-|-|-|-| | metrics | map[string][AnalysisMetrics](/docs/user-guide/configuration-reference/#analysismetrics) | Template for metrics. | No | -## Image Watcher Configuration - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: ImageWatcher -spec: - targets: - - image: gcr.io/pipecd/helloworld - provider: my-gcr - filePath: foo/deployment.yaml - field: $.spec.template.spec.containers[0].image -``` - -| Field | Type | Description | Required | -|-|-|-|-| -| targets | [][ImageWatcherTarget](/docs/user-guide/configuration-reference/#imagewatchertarget) | Target images to be watched. | No | - -## ImageWatcherTarget - -| Field | Type | Description | Required | -|-|-|-|-| -| image | string | Fully qualified image name. | Yes | -| provider | string | The name of Image provider that must be configured in the piped config. See [here](/docs/operator-manual/piped/configuration-reference/#imageprovider) for more details. | Yes | -| filePath | string | The path to the file to be updated. | Yes | -| field | string | The path to the field to be updated. It requires to start with `$` which represents the root element. e.g. `$.foo.bar[0].baz`. | Yes | - ## Event Watcher Configuration ```yaml diff --git a/docs/content/en/docs/user-guide/image-watcher.md b/docs/content/en/docs/user-guide/image-watcher.md deleted file mode 100644 index d42efb6ce8..0000000000 --- a/docs/content/en/docs/user-guide/image-watcher.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: "Image watcher" -linkTitle: "Image watcher" -weight: 12 -description: > - Watching container image changes and automatically deploying the new images. ---- - -Image watcher automatically triggers a new Deployment when a new image tag is pushed to your container registry. - -The canonical deployment flow with PipeCD is: - -1. CI server pushes the generated image to the container registry after app-repo updated. -1. You update the config-repo manually. - -It is the User's responsibility to automate these steps to be done in a series of actions, while it is quite a bit of painful. -Image watcher lets you automate this workflow by continuously performing `git push` to your config-repo. -That is, it frees you from the hassle of manually updating config-repo every time. - -## Prerequisites -Before configuring ImageWatcher, all required Image providers must be configured in the Piped Configuration according to [this guide](/docs/operator-manual/piped/configuring-image-watcher/). - -## Configuration - -Prepare ImageWatcher files placed at the `.pipe/` directory at the root of the Git repository. -In that files, you define what image should be watched and what file should be updated. - -```yaml -apiVersion: pipecd.dev/v1beta1 -kind: ImageWatcher -spec: - targets: - - image: gcr.io/pipecd/helloworld - provider: my-gcr - filePath: helloworld/deployment.yaml - field: $.spec.template.spec.containers[0].image -``` - -Image watcher periodically compares the latest tag of the following two images: -- a given `image` in a given `provider` -- an image defined at a given `field` in a given `filePath` - -And then pushes them to the git repository if there are any deviations. -Note that it uses only pure git push, does not use features that depend on Git hosting services, such as pull requests. - - -### Examples -Suppose there is the above ImageWatcher file placed at the `.pipe/` directory at the root of the Git repository, and the below kubernetes manifest is placed at `helloworld/deployment.yaml`. - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - ... -spec: - ... - template: - spec: - containers: - - name: helloworld - image: gcr.io/pipecd/helloworld:0.1.0 -``` - -Then let's say you pushed the new tag `0.2.0` to the image provider with name `my-gcr`. Piped will create a commit as shown below: - -```diff - spec: - containers: - - name: helloworld -- image: gcr.io/pipecd/helloworld:0.1.0 -+ image: gcr.io/pipecd/helloworld:0.2.0 -``` - -See [here](https://github.com/pipe-cd/examples/tree/master/.pipe) for more examples. - -The full list of configurable `ImageWatcher` fields are [here](/docs/user-guide/configuration-reference/#image-watcher-configuration). - ->ProTip: If multiple Pipeds handle a single repository, you can prevent conflicts by splitting into the multiple files and specifying includes/excludes in the Piped config. diff --git a/examples/.pipe/image-watcher.yaml b/examples/.pipe/image-watcher.yaml deleted file mode 100644 index d81fa7bb0f..0000000000 --- a/examples/.pipe/image-watcher.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: pipecd.dev/v1beta1 -kind: ImageWatcher -spec: - targets: - - image: gcr.io/pipecd/helloworld - provider: dev-gcr - filePath: kubernetes/simple/deployment.yaml - field: $.spec.template.spec.containers[0].image - - image: gcr.io/pipecd/helloworld - provider: dev-gcr - filePath: cloudrun/simple/service.yaml - field: $.spec.template.spec.containers[0].image diff --git a/pkg/app/piped/cmd/piped/BUILD.bazel b/pkg/app/piped/cmd/piped/BUILD.bazel index cb6597b5d1..99ce63a0b0 100644 --- a/pkg/app/piped/cmd/piped/BUILD.bazel +++ b/pkg/app/piped/cmd/piped/BUILD.bazel @@ -19,7 +19,6 @@ go_library( "//pkg/app/piped/driftdetector:go_default_library", "//pkg/app/piped/eventwatcher:go_default_library", "//pkg/app/piped/executor/registry:go_default_library", - "//pkg/app/piped/imagewatcher:go_default_library", "//pkg/app/piped/livestatereporter:go_default_library", "//pkg/app/piped/livestatestore:go_default_library", "//pkg/app/piped/notifier:go_default_library", diff --git a/pkg/app/piped/cmd/piped/piped.go b/pkg/app/piped/cmd/piped/piped.go index 32b7159e2b..564da8b6b6 100644 --- a/pkg/app/piped/cmd/piped/piped.go +++ b/pkg/app/piped/cmd/piped/piped.go @@ -39,7 +39,6 @@ import ( "github.com/pipe-cd/pipe/pkg/app/piped/controller" "github.com/pipe-cd/pipe/pkg/app/piped/driftdetector" "github.com/pipe-cd/pipe/pkg/app/piped/eventwatcher" - "github.com/pipe-cd/pipe/pkg/app/piped/imagewatcher" "github.com/pipe-cd/pipe/pkg/app/piped/livestatereporter" "github.com/pipe-cd/pipe/pkg/app/piped/livestatestore" "github.com/pipe-cd/pipe/pkg/app/piped/notifier" @@ -339,18 +338,6 @@ func (p *piped) run(ctx context.Context, t cli.Telemetry) (runErr error) { }) } - if len(cfg.ImageProviders) > 0 { - // Start running image watcher. - t := imagewatcher.NewWatcher( - cfg, - gitClient, - t.Logger, - ) - group.Go(func() error { - return t.Run(ctx) - }) - } - { // Start running event watcher. t := eventwatcher.NewWatcher( diff --git a/pkg/app/piped/imageprovider/BUILD.bazel b/pkg/app/piped/imageprovider/BUILD.bazel deleted file mode 100644 index 29a186b94d..0000000000 --- a/pkg/app/piped/imageprovider/BUILD.bazel +++ /dev/null @@ -1,14 +0,0 @@ -load("@io_bazel_rules_go//go:def.bzl", "go_library") - -go_library( - name = "go_default_library", - srcs = ["provider.go"], - importpath = "github.com/pipe-cd/pipe/pkg/app/piped/imageprovider", - visibility = ["//visibility:public"], - deps = [ - "//pkg/app/piped/imageprovider/ecr:go_default_library", - "//pkg/config:go_default_library", - "//pkg/model:go_default_library", - "@org_uber_go_zap//:go_default_library", - ], -) diff --git a/pkg/app/piped/imageprovider/ecr/BUILD.bazel b/pkg/app/piped/imageprovider/ecr/BUILD.bazel deleted file mode 100644 index 1a5fd40682..0000000000 --- a/pkg/app/piped/imageprovider/ecr/BUILD.bazel +++ /dev/null @@ -1,26 +0,0 @@ -load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") - -go_library( - name = "go_default_library", - srcs = ["ecr.go"], - importpath = "github.com/pipe-cd/pipe/pkg/app/piped/imageprovider/ecr", - visibility = ["//visibility:public"], - deps = [ - "//pkg/model:go_default_library", - "@com_github_aws_aws_sdk_go//aws:go_default_library", - "@com_github_aws_aws_sdk_go//aws/awserr:go_default_library", - "@com_github_aws_aws_sdk_go//aws/credentials:go_default_library", - "@com_github_aws_aws_sdk_go//aws/credentials/ec2rolecreds:go_default_library", - "@com_github_aws_aws_sdk_go//aws/ec2metadata:go_default_library", - "@com_github_aws_aws_sdk_go//aws/session:go_default_library", - "@com_github_aws_aws_sdk_go//service/ecr:go_default_library", - "@org_uber_go_zap//:go_default_library", - ], -) - -go_test( - name = "go_default_test", - size = "small", - srcs = ["ecr_test.go"], - embed = [":go_default_library"], -) diff --git a/pkg/app/piped/imageprovider/ecr/ecr.go b/pkg/app/piped/imageprovider/ecr/ecr.go deleted file mode 100644 index ea6309b884..0000000000 --- a/pkg/app/piped/imageprovider/ecr/ecr.go +++ /dev/null @@ -1,196 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package ecr - -import ( - "context" - "fmt" - "sort" - "strings" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/awserr" - "github.com/aws/aws-sdk-go/aws/credentials" - "github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds" - "github.com/aws/aws-sdk-go/aws/ec2metadata" - "github.com/aws/aws-sdk-go/aws/session" - "github.com/aws/aws-sdk-go/service/ecr" - "go.uber.org/zap" - - "github.com/pipe-cd/pipe/pkg/model" -) - -// The maximum number of image results returned by the APIs about images. -// The API allows this value to be between 1 and 1000. -// See more: https://pkg.go.dev/github.com/aws/aws-sdk-go/service/ecr#ListImagesInput -const maxResults = 1000 - -type ECR struct { - name string - client *ecr.ECR - region string - credentialsFile string - profile string - registryID string - - logger *zap.Logger -} - -type Option func(*ECR) - -func WithRegistryID(id string) Option { - return func(e *ECR) { - e.registryID = id - } -} - -func WithCredentialsFile(path string) Option { - return func(e *ECR) { - e.credentialsFile = path - } -} - -func WithProfile(profile string) Option { - return func(e *ECR) { - e.profile = profile - } -} - -func WithLogger(logger *zap.Logger) Option { - return func(e *ECR) { - e.logger = logger - } -} - -// NewECR attempts to retrieve credentials in the following order: -// 1. from the environment variables. Available environment variables are: -// - AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY -// - AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY -// 2. from the given credentials file. -// 3. from the EC2 Instance Role -func NewECR(name string, region string, opts ...Option) (*ECR, error) { - if region == "" { - return nil, fmt.Errorf("region is required") - } - e := &ECR{ - name: name, - region: region, - logger: zap.NewNop(), - } - for _, opt := range opts { - opt(e) - } - e.logger = e.logger.Named("ecr-provider") - - sess, err := session.NewSession() - if err != nil { - return nil, fmt.Errorf("failed to create a session: %w", err) - } - creds := credentials.NewChainCredentials( - []credentials.Provider{ - &credentials.EnvProvider{}, - &credentials.SharedCredentialsProvider{ - Filename: e.credentialsFile, - Profile: e.profile, - }, - &ec2rolecreds.EC2RoleProvider{ - Client: ec2metadata.New(sess), - }, - }, - ) - cfg := aws.NewConfig().WithRegion(e.region).WithCredentials(creds) - // TODO: Use ecrpublic package when public image given - // See more: https://docs.aws.amazon.com/sdk-for-go/api/service/ecrpublic - e.client = ecr.New(sess, cfg) - return e, nil -} - -func (e *ECR) Name() string { - return e.name -} - -func (e *ECR) Type() model.ImageProviderType { - return model.ImageProviderTypeECR -} - -func (e *ECR) ParseImage(image string) (*model.ImageName, error) { - ss := strings.SplitN(image, "/", 2) - if len(ss) < 2 { - return nil, fmt.Errorf("invalid image format (e.g. account-id.dkr.ecr.region.amazon.aws.com/pipecd/helloworld)") - } - return &model.ImageName{ - Domain: ss[0], - Repo: ss[1], - }, nil -} - -func (e *ECR) GetLatestImage(ctx context.Context, image *model.ImageName) (*model.ImageRef, error) { - input := &ecr.DescribeImagesInput{ - RepositoryName: aws.String(image.Repo), - Filter: &ecr.DescribeImagesFilter{TagStatus: aws.String("TAGGED")}, - MaxResults: aws.Int64(maxResults), - } - if e.registryID != "" { - input.RegistryId = &e.registryID - } - - // TODO: Consider the way to determine the latest tag other than fetching all tags - // - // Iterate over the pages of a DescribeImages operation until the last page. - // NOTE: A lot of requests may be issued if there are a lot of tags. - // For instance, for 6k tags, it will issue 6 requests. - imageDetails := make([]*ecr.ImageDetail, 0, maxResults) - err := e.client.DescribeImagesPagesWithContext(ctx, input, func(page *ecr.DescribeImagesOutput, lastPage bool) bool { - imageDetails = append(imageDetails, page.ImageDetails...) - return true - }) - if err != nil { - if aerr, ok := err.(awserr.Error); ok { - switch aerr.Code() { - case ecr.ErrCodeServerException: - return nil, fmt.Errorf("server-side issue occured: %w", err) - case ecr.ErrCodeInvalidParameterException: - return nil, fmt.Errorf("invalid parameter given: %w", err) - case ecr.ErrCodeRepositoryNotFoundException: - return nil, fmt.Errorf("repository not found: %w", err) - case ecr.ErrCodeImageNotFoundException: - return nil, fmt.Errorf("image not found: %w", err) - } - } - return nil, fmt.Errorf("unknow error given: %w", err) - } - if len(imageDetails) == 0 { - return nil, fmt.Errorf("no images found") - } - sort.Slice(imageDetails, func(i, j int) bool { - l, r := imageDetails[i], imageDetails[j] - if l.ImagePushedAt == nil || r.ImagePushedAt == nil { - return l.ImagePushedAt == nil && r.ImagePushedAt != nil - } - return l.ImagePushedAt.After(*r.ImagePushedAt) - }) - if len(imageDetails[0].ImageTags) == 0 { - return nil, fmt.Errorf("no images tag is associated the image") - } - // NOTE: Even if the tags are different, they are managed as a single - // image if the images' sha256 digests are identical, so there may - // be multiple tags associated with a single image. That's why - // an ImageDetail has multiple tags. - latest := *imageDetails[0].ImageTags[0] - return &model.ImageRef{ - ImageName: *image, - Tag: latest, - }, nil -} diff --git a/pkg/app/piped/imageprovider/ecr/ecr_test.go b/pkg/app/piped/imageprovider/ecr/ecr_test.go deleted file mode 100644 index 4a0540f8f8..0000000000 --- a/pkg/app/piped/imageprovider/ecr/ecr_test.go +++ /dev/null @@ -1,15 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package ecr diff --git a/pkg/app/piped/imageprovider/gcr/BUILD.bazel b/pkg/app/piped/imageprovider/gcr/BUILD.bazel deleted file mode 100644 index 385cef7a25..0000000000 --- a/pkg/app/piped/imageprovider/gcr/BUILD.bazel +++ /dev/null @@ -1,8 +0,0 @@ -load("@io_bazel_rules_go//go:def.bzl", "go_library") - -go_library( - name = "go_default_library", - srcs = ["gcr.go"], - importpath = "github.com/pipe-cd/pipe/pkg/app/piped/imageprovider/gcr", - visibility = ["//visibility:public"], -) diff --git a/pkg/app/piped/imageprovider/gcr/gcr.go b/pkg/app/piped/imageprovider/gcr/gcr.go deleted file mode 100644 index 39cbccd5a2..0000000000 --- a/pkg/app/piped/imageprovider/gcr/gcr.go +++ /dev/null @@ -1,137 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package gcr - -/* -import ( - "context" - "fmt" - "io/ioutil" - "sort" - "strings" - - "github.com/google/go-containerregistry/pkg/authn" - "github.com/google/go-containerregistry/pkg/name" - "github.com/google/go-containerregistry/pkg/v1/google" - "go.uber.org/zap" - - "github.com/pipe-cd/pipe/pkg/model" -) - -type GCR struct { - name string - serviceAccountFile string - // If nil, treated as an anonymous user. - authenticator authn.Authenticator - logger *zap.Logger -} - -type Option func(*GCR) - -func WithServiceAccountFile(path string) Option { - return func(e *GCR) { - e.serviceAccountFile = path - } -} - -func WithLogger(logger *zap.Logger) Option { - return func(e *GCR) { - e.logger = logger - } -} - -// NewGCR generates a GCR client with an anonymous user if no authenticate method set. -func NewGCR(name string, opts ...Option) (*GCR, error) { - g := &GCR{ - name: name, - logger: zap.NewNop(), - } - for _, opt := range opts { - opt(g) - } - g.logger = g.logger.Named("gcr-provider") - - if g.serviceAccountFile != "" { - b, err := ioutil.ReadFile(g.serviceAccountFile) - if err != nil { - return nil, fmt.Errorf("failed to open the service account file: %w", err) - } - g.authenticator = google.NewJSONKeyAuthenticator(string(b)) - } - return g, nil -} - -func (g *GCR) Name() string { - return g.name -} - -func (g *GCR) Type() model.ImageProviderType { - return model.ImageProviderTypeGCR -} - -func (g *GCR) ParseImage(image string) (*model.ImageName, error) { - ss := strings.SplitN(image, "/", 2) - if len(ss) < 2 { - return nil, fmt.Errorf("invalid image format (e.g. gcr.io/pipecd/helloworld)") - } - return &model.ImageName{ - Domain: ss[0], - Repo: ss[1], - }, nil -} - -func (g *GCR) GetLatestImage(ctx context.Context, image *model.ImageName) (*model.ImageRef, error) { - repo, err := name.NewRepository(image.String()) - if err != nil { - return nil, fmt.Errorf("%s is invalid repository: %w", image, err) - } - options := []google.ListerOption{ - google.WithContext(ctx), - } - if g.authenticator != nil { - options = append(options, google.WithAuth(g.authenticator)) - } - // TODO: Use pagination to retrieve image tags from GCR - // Currently, the result could be quite large size if there are a lot of tags. - // "google/go-containerregistry" doesn't provide any option to paginate. - // We can propose it to them, or just borrow and modify for us. - // See more: https://docs.docker.com/registry/spec/api/#listing-image-tags - res, err := google.List(repo, options...) - if err != nil { - return nil, fmt.Errorf("failed to list tags: %w", err) - } - if len(res.Manifests) == 0 { - return nil, fmt.Errorf("no manifests found in %s", repo.Name()) - } - - // Determine the latest by sorting by the uploaded time. - manifests := make([]google.ManifestInfo, 0, len(res.Manifests)) - for _, m := range res.Manifests { - manifests = append(manifests, m) - } - sort.Slice(manifests, func(i, j int) bool { - return manifests[i].Uploaded.After(manifests[j].Uploaded) - }) - latest := manifests[0] - if len(latest.Tags) == 0 { - return nil, fmt.Errorf("no tag is associated to the latest image") - } - return &model.ImageRef{ - ImageName: *image, - // TODO: Enable to specify the tag if multiple tags are associated to an image - Tag: latest.Tags[0], - }, nil -} -*/ diff --git a/pkg/app/piped/imageprovider/provider.go b/pkg/app/piped/imageprovider/provider.go deleted file mode 100644 index 3844dc2a74..0000000000 --- a/pkg/app/piped/imageprovider/provider.go +++ /dev/null @@ -1,65 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package imageprovider - -import ( - "context" - "fmt" - - "go.uber.org/zap" - - "github.com/pipe-cd/pipe/pkg/app/piped/imageprovider/ecr" - "github.com/pipe-cd/pipe/pkg/config" - "github.com/pipe-cd/pipe/pkg/model" -) - -// Provider acts as a container registry client. -type Provider interface { - // Name gives back the provider name that is unique in the Piped. - Name() string - // Type indicates which container registry client to act as. - Type() model.ImageProviderType - // ParseImage converts the given string into structured image. - ParseImage(image string) (*model.ImageName, error) - // GetLatestImages gives back an image with the latest tag. - GetLatestImage(ctx context.Context, image *model.ImageName) (*model.ImageRef, error) -} - -// NewProvider yields an appropriate provider according to the given config. -func NewProvider(cfg *config.PipedImageProvider, logger *zap.Logger) (Provider, error) { - switch cfg.Type { - case model.ImageProviderTypeGCR: - /* - options := []gcr.Option{ - gcr.WithServiceAccountFile(cfg.GCRConfig.ServiceAccountFile), - gcr.WithLogger(logger), - } - return gcr.NewGCR(cfg.Name, options...) - */ - return nil, nil - case model.ImageProviderTypeECR: - options := []ecr.Option{ - ecr.WithRegistryID(cfg.ECRConfig.RegistryID), - ecr.WithCredentialsFile(cfg.ECRConfig.CredentialsFile), - ecr.WithProfile(cfg.ECRConfig.Profile), - ecr.WithLogger(logger), - } - return ecr.NewECR(cfg.Name, cfg.ECRConfig.Region, options...) - case model.ImageProviderTypeDockerHub: - return nil, fmt.Errorf("not implemented yet") - default: - return nil, fmt.Errorf("unknown image provider type: %s", cfg.Type) - } -} diff --git a/pkg/app/piped/imagewatcher/BUILD.bazel b/pkg/app/piped/imagewatcher/BUILD.bazel deleted file mode 100644 index 3a068049a5..0000000000 --- a/pkg/app/piped/imagewatcher/BUILD.bazel +++ /dev/null @@ -1,15 +0,0 @@ -load("@io_bazel_rules_go//go:def.bzl", "go_library") - -go_library( - name = "go_default_library", - srcs = ["watcher.go"], - importpath = "github.com/pipe-cd/pipe/pkg/app/piped/imagewatcher", - visibility = ["//visibility:public"], - deps = [ - "//pkg/app/piped/imageprovider:go_default_library", - "//pkg/config:go_default_library", - "//pkg/git:go_default_library", - "//pkg/yamlprocessor:go_default_library", - "@org_uber_go_zap//:go_default_library", - ], -) diff --git a/pkg/app/piped/imagewatcher/watcher.go b/pkg/app/piped/imagewatcher/watcher.go deleted file mode 100644 index c7afeb4d75..0000000000 --- a/pkg/app/piped/imagewatcher/watcher.go +++ /dev/null @@ -1,262 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Package imagewatcher provides a piped component -// that periodically checks the container registry and updates -// the image if there are differences with Git. -package imagewatcher - -import ( - "context" - "errors" - "fmt" - "io/ioutil" - "os" - "path/filepath" - "sync" - "time" - - "go.uber.org/zap" - - "github.com/pipe-cd/pipe/pkg/app/piped/imageprovider" - "github.com/pipe-cd/pipe/pkg/config" - "github.com/pipe-cd/pipe/pkg/git" - "github.com/pipe-cd/pipe/pkg/yamlprocessor" -) - -const ( - // image ref, file name and new tag are supposed. - defaultCommitMessageFormat = "Update %s in %s to %s" - defaultCheckInterval = 5 * time.Minute -) - -type Watcher interface { - Run(context.Context) error -} - -type gitClient interface { - Clone(ctx context.Context, repoID, remote, branch, destination string) (git.Repo, error) -} - -type commit struct { - changes map[string][]byte - message string -} - -type watcher struct { - config *config.PipedSpec - gitClient gitClient - logger *zap.Logger - wg sync.WaitGroup - - // Indexed by the Image Provider name. - providerCfgs map[string]config.PipedImageProvider -} - -func NewWatcher(cfg *config.PipedSpec, gitClient gitClient, logger *zap.Logger) Watcher { - return &watcher{ - config: cfg, - gitClient: gitClient, - logger: logger.Named("image-watcher"), - } -} - -// Run spawns goroutines for each git repository. They periodically pull the image -// from the container registry to compare the image with one in the git repository. -func (w *watcher) Run(ctx context.Context) error { - w.logger.Info("start running image watcher") - - w.providerCfgs = make(map[string]config.PipedImageProvider, len(w.config.ImageProviders)) - for _, cfg := range w.config.ImageProviders { - w.providerCfgs[cfg.Name] = cfg - } - - for _, repoCfg := range w.config.Repositories { - repo, err := w.gitClient.Clone(ctx, repoCfg.RepoID, repoCfg.Remote, repoCfg.Branch, "") - if err != nil { - w.logger.Error("failed to clone repository", - zap.String("repo-id", repoCfg.RepoID), - zap.Error(err), - ) - return fmt.Errorf("failed to clone repository %s: %w", repoCfg.RepoID, err) - } - defer os.RemoveAll(repo.GetPath()) - - w.wg.Add(1) - go w.run(ctx, repo, &repoCfg) - } - - w.wg.Wait() - return nil -} - -// run periodically compares the image in the given git repository and one in the image provider. -// And then pushes those with differences. -func (w *watcher) run(ctx context.Context, repo git.Repo, repoCfg *config.PipedRepository) { - defer w.wg.Done() - - var ( - commitMsg string - includedCfgs, excludedCfgs []string - ) - // Use user-defined settings if there is. - for _, r := range w.config.ImageWatcher.GitRepos { - if r.RepoID != repoCfg.RepoID { - continue - } - commitMsg = r.CommitMessage - includedCfgs = r.Includes - excludedCfgs = r.Excludes - break - } - checkInterval := time.Duration(w.config.ImageWatcher.CheckInterval) - if checkInterval == 0 { - checkInterval = defaultCheckInterval - } - - ticker := time.NewTicker(checkInterval) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - err := repo.Pull(ctx, repo.GetClonedBranch()) - if err != nil { - w.logger.Error("failed to perform git pull", - zap.String("repo-id", repoCfg.RepoID), - zap.String("branch", repo.GetClonedBranch()), - zap.Error(err), - ) - continue - } - cfg, err := config.LoadImageWatcher(repo.GetPath(), includedCfgs, excludedCfgs) - if errors.Is(err, config.ErrNotFound) { - w.logger.Info("configuration file for Image Watcher not found", - zap.String("repo-id", repoCfg.RepoID), - zap.Error(err), - ) - continue - } - if err != nil { - w.logger.Error("failed to load configuration file for Image Watcher", - zap.String("repo-id", repoCfg.RepoID), - zap.Error(err), - ) - continue - } - if err := w.updateOutdatedImages(ctx, repo, cfg.Targets, commitMsg); err != nil { - w.logger.Error("failed to update the targets", - zap.String("repo-id", repoCfg.RepoID), - zap.Error(err), - ) - } - } - } -} - -// updateOutdatedImages inspects all targets and pushes the changes to git repo if there is. -func (w *watcher) updateOutdatedImages(ctx context.Context, repo git.Repo, targets []config.ImageWatcherTarget, commitMsg string) error { - commits := make([]*commit, 0) - for _, t := range targets { - c, err := w.checkOutdatedImage(ctx, &t, repo, commitMsg) - if err != nil { - w.logger.Error("failed to update image", zap.Error(err)) - continue - } - if c != nil { - commits = append(commits, c) - } - } - if len(commits) == 0 { - return nil - } - - w.logger.Info(fmt.Sprintf("there are %d outdated images", len(commits))) - // Copy the repo to another directory to avoid pull failure in the future. - tmpDir, err := ioutil.TempDir("", "image-watcher") - if err != nil { - return fmt.Errorf("failed to create a new temporary directory: %w", err) - } - defer os.RemoveAll(tmpDir) - tmpRepo, err := repo.Copy(filepath.Join(tmpDir, "tmp-repo")) - if err != nil { - return fmt.Errorf("failed to copy the repository to the temporary directory: %w", err) - } - for _, c := range commits { - if err := tmpRepo.CommitChanges(ctx, tmpRepo.GetClonedBranch(), c.message, false, c.changes); err != nil { - return fmt.Errorf("failed to perform git commit: %w", err) - } - } - - return tmpRepo.Push(ctx, tmpRepo.GetClonedBranch()) -} - -// checkOutdatedImage gives back a change content if any deviation exists -// between the image in the given git repository and one in the image provider. -func (w *watcher) checkOutdatedImage(ctx context.Context, target *config.ImageWatcherTarget, repo git.Repo, commitMsg string) (*commit, error) { - // Retrieve the image from the image provider. - providerCfg, ok := w.providerCfgs[target.Provider] - if !ok { - return nil, fmt.Errorf("unknown image provider %s is defined", target.Provider) - } - provider, err := imageprovider.NewProvider(&providerCfg, w.logger) - if err != nil { - return nil, fmt.Errorf("failed to yield image provider %s: %w", providerCfg.Name, err) - } - i, err := provider.ParseImage(target.Image) - if err != nil { - return nil, fmt.Errorf("failed to parse image string \"%s\": %w", target.Image, err) - } - // TODO: Control not to reach the rate limit - imageInRegistry, err := provider.GetLatestImage(ctx, i) - if err != nil { - return nil, fmt.Errorf("failed to get latest image from %s: %w", provider.Name(), err) - } - - // Retrieve the image from the file cloned from the git repository. - path := filepath.Join(repo.GetPath(), target.FilePath) - yml, err := ioutil.ReadFile(path) - if err != nil { - return nil, fmt.Errorf("failed to read file: %w", err) - } - value, err := yamlprocessor.GetValue(yml, target.Field) - if err != nil { - return nil, fmt.Errorf("failed to get value at %s in %s: %w", target.Field, target.FilePath, err) - } - imageInGit, ok := value.(string) - if !ok { - return nil, fmt.Errorf("unknown value is defined at %s in %s", target.FilePath, target.Field) - } - - outdated := imageInRegistry.String() != imageInGit - if !outdated { - return nil, nil - } - - // Give back a change content. - newYml, err := yamlprocessor.ReplaceValue(yml, target.Field, imageInRegistry.String()) - if err != nil { - return nil, fmt.Errorf("failed to replace value at %s with %s: %w", target.Field, imageInRegistry, err) - } - if commitMsg == "" { - commitMsg = fmt.Sprintf(defaultCommitMessageFormat, imageInGit, target.FilePath, imageInRegistry.Tag) - } - return &commit{ - changes: map[string][]byte{ - target.FilePath: newYml, - }, - message: commitMsg, - }, nil -} diff --git a/pkg/config/BUILD.bazel b/pkg/config/BUILD.bazel index 5f9a7efda0..7303de8245 100644 --- a/pkg/config/BUILD.bazel +++ b/pkg/config/BUILD.bazel @@ -14,7 +14,6 @@ go_library( "deployment_terraform.go", "duration.go", "event_watcher.go", - "image_watcher.go", "piped.go", "replicas.go", "sealed_secret.go", @@ -42,7 +41,6 @@ go_test( "deployment_terraform_test.go", "deployment_test.go", "event_watcher_test.go", - "image_watcher_test.go", "piped_test.go", "replicas_test.go", "sealed_secret_test.go", diff --git a/pkg/config/config.go b/pkg/config/config.go index 54ff79162c..d487b33b36 100644 --- a/pkg/config/config.go +++ b/pkg/config/config.go @@ -62,8 +62,6 @@ const ( // This configuration file should be placed in .pipe directory // at the root of the repository. KindAnalysisTemplate Kind = "AnalysisTemplate" - // KindImageWatcher represents configuration for Repo Watcher. - KindImageWatcher Kind = "ImageWatcher" // KindEventWatcher represents configuration for Event Watcher. KindEventWatcher Kind = "EventWatcher" ) @@ -88,7 +86,6 @@ type Config struct { PipedSpec *PipedSpec ControlPlaneSpec *ControlPlaneSpec AnalysisTemplateSpec *AnalysisTemplateSpec - ImageWatcherSpec *ImageWatcherSpec EventWatcherSpec *EventWatcherSpec SealedSecretSpec *SealedSecretSpec @@ -149,10 +146,6 @@ func (c *Config) init(kind Kind, apiVersion string) error { c.SealedSecretSpec = &SealedSecretSpec{} c.spec = c.SealedSecretSpec - case KindImageWatcher: - c.ImageWatcherSpec = &ImageWatcherSpec{} - c.spec = c.ImageWatcherSpec - case KindEventWatcher: c.EventWatcherSpec = &EventWatcherSpec{} c.spec = c.EventWatcherSpec diff --git a/pkg/config/image_watcher.go b/pkg/config/image_watcher.go deleted file mode 100644 index aa69db82d1..0000000000 --- a/pkg/config/image_watcher.go +++ /dev/null @@ -1,137 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package config - -import ( - "fmt" - "io/ioutil" - "os" - "path/filepath" -) - -type ImageWatcherSpec struct { - Targets []ImageWatcherTarget `json:"targets"` -} - -// ImageWatcherTarget provides information to compare the latest tags. -// -// Image watcher typically compares the "Image" in the "Provider" and -// an image defined at the "Field" in the "FilePath". -type ImageWatcherTarget struct { - // The name of Image provider. - Provider string `json:"provider"` - // Fully qualified image name. - Image string `json:"image"` - // The path to the file to be updated. - FilePath string `json:"filePath"` - // The path to the field to be updated. It requires to start - // with `$` which represents the root element. e.g. `$.foo.bar[0].baz`. - Field string `json:"field"` -} - -// LoadImageWatcher finds the config files for the image watcher in the .pipe -// directory first up. And returns parsed config after merging the targets. -// Only one of includes or excludes can be used. ErrNotFound is returned if not found. -func LoadImageWatcher(repoRoot string, includes, excludes []string) (*ImageWatcherSpec, error) { - dir := filepath.Join(repoRoot, SharedConfigurationDirName) - files, err := ioutil.ReadDir(dir) - if os.IsNotExist(err) { - return nil, ErrNotFound - } - if err != nil { - return nil, fmt.Errorf("failed to read %s: %w", dir, err) - } - - spec := &ImageWatcherSpec{ - Targets: make([]ImageWatcherTarget, 0), - } - filtered, err := filterImageWatcherFiles(files, includes, excludes) - if err != nil { - return nil, fmt.Errorf("failed to filter image watcher files at %s: %w", dir, err) - } - for _, f := range filtered { - if f.IsDir() { - continue - } - path := filepath.Join(dir, f.Name()) - cfg, err := LoadFromYAML(path) - if err != nil { - return nil, fmt.Errorf("failed to load config file %s: %w", path, err) - } - if cfg.Kind == KindImageWatcher { - spec.Targets = append(spec.Targets, cfg.ImageWatcherSpec.Targets...) - } - } - if len(spec.Targets) == 0 { - return nil, ErrNotFound - } - if err := spec.Validate(); err != nil { - return nil, err - } - - return spec, nil -} - -// filterImageWatcherFiles filters the given files based on the given Includes and Excludes. -// Excludes are prioritized if both Excludes and Includes are given. -func filterImageWatcherFiles(files []os.FileInfo, includes, excludes []string) ([]os.FileInfo, error) { - if len(includes) == 0 && len(excludes) == 0 { - return files, nil - } - - filtered := make([]os.FileInfo, 0, len(files)) - useWhitelist := len(includes) != 0 && len(excludes) == 0 - if useWhitelist { - whiteList := make(map[string]struct{}, len(includes)) - for _, i := range includes { - whiteList[i] = struct{}{} - } - for _, f := range files { - if _, ok := whiteList[f.Name()]; ok { - filtered = append(filtered, f) - } - } - return filtered, nil - } - - blackList := make(map[string]struct{}, len(excludes)) - for _, e := range excludes { - blackList[e] = struct{}{} - } - for _, f := range files { - if _, ok := blackList[f.Name()]; !ok { - filtered = append(filtered, f) - } - } - return filtered, nil -} - -func (s *ImageWatcherSpec) Validate() error { - for _, t := range s.Targets { - if t.Provider == "" { - return fmt.Errorf("provider must not be empty") - } - if t.Image == "" { - return fmt.Errorf("image must not be empty") - } - if t.FilePath == "" { - return fmt.Errorf("filePath must not be empty") - } - if t.Field == "" { - return fmt.Errorf("field must not be empty") - } - } - return nil -} diff --git a/pkg/config/image_watcher_test.go b/pkg/config/image_watcher_test.go deleted file mode 100644 index 7698e3c2b3..0000000000 --- a/pkg/config/image_watcher_test.go +++ /dev/null @@ -1,183 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package config - -import ( - "os" - "testing" - - "github.com/stretchr/testify/assert" -) - -func TestFilterImageWatcherFiles(t *testing.T) { - testcases := []struct { - name string - files []os.FileInfo - includes []string - excludes []string - want []os.FileInfo - wantErr bool - }{ - { - name: "both includes and excludes aren't given", - files: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - }, - want: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - }, - wantErr: false, - }, - { - name: "both includes and excludes are given", - files: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - }, - want: []os.FileInfo{}, - includes: []string{"file-1"}, - excludes: []string{"file-1"}, - wantErr: false, - }, - { - name: "includes given", - files: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - &fakeFileInfo{ - name: "file-2", - }, - &fakeFileInfo{ - name: "file-3", - }, - }, - includes: []string{"file-1", "file-3"}, - want: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - &fakeFileInfo{ - name: "file-3", - }, - }, - wantErr: false, - }, - { - name: "excludes given", - files: []os.FileInfo{ - &fakeFileInfo{ - name: "file-1", - }, - &fakeFileInfo{ - name: "file-2", - }, - &fakeFileInfo{ - name: "file-3", - }, - }, - excludes: []string{"file-1", "file-3"}, - want: []os.FileInfo{ - &fakeFileInfo{ - name: "file-2", - }, - }, - wantErr: false, - }, - } - for _, tc := range testcases { - t.Run(tc.name, func(t *testing.T) { - got, err := filterImageWatcherFiles(tc.files, tc.includes, tc.excludes) - assert.Equal(t, tc.wantErr, err != nil) - assert.Equal(t, tc.want, got) - }) - } -} - -func TestImageWatcherValidate(t *testing.T) { - testcases := []struct { - name string - imageWatcherSpec ImageWatcherSpec - wantErr bool - }{ - { - name: "empty provider given", - imageWatcherSpec: ImageWatcherSpec{Targets: []ImageWatcherTarget{ - { - Image: "image", - FilePath: "filePath", - Field: "field", - }, - }}, - wantErr: true, - }, - { - name: "empty image given", - imageWatcherSpec: ImageWatcherSpec{Targets: []ImageWatcherTarget{ - { - Provider: "provider", - FilePath: "filePath", - Field: "field", - }, - }}, - wantErr: true, - }, - { - name: "empty file path given", - imageWatcherSpec: ImageWatcherSpec{Targets: []ImageWatcherTarget{ - { - Provider: "provider", - Image: "image", - Field: "field", - }, - }}, - wantErr: true, - }, - { - name: "empty field given", - imageWatcherSpec: ImageWatcherSpec{Targets: []ImageWatcherTarget{ - { - Provider: "provider", - Image: "image", - FilePath: "filePath", - }, - }}, - wantErr: true, - }, - { - name: "all fields given", - imageWatcherSpec: ImageWatcherSpec{Targets: []ImageWatcherTarget{ - { - Provider: "provider", - Image: "image", - FilePath: "filePath", - Field: "field", - }, - }}, - wantErr: false, - }, - } - for _, tc := range testcases { - t.Run(tc.name, func(t *testing.T) { - err := tc.imageWatcherSpec.Validate() - assert.Equal(t, tc.wantErr, err != nil) - }) - } -} diff --git a/pkg/config/piped.go b/pkg/config/piped.go index 8e68c3a00f..466850dc72 100644 --- a/pkg/config/piped.go +++ b/pkg/config/piped.go @@ -53,14 +53,10 @@ type PipedSpec struct { CloudProviders []PipedCloudProvider `json:"cloudProviders"` // List of analysis providers can be used by this piped. AnalysisProviders []PipedAnalysisProvider `json:"analysisProviders"` - // List of image providers can be used by this piped. - ImageProviders []PipedImageProvider `json:"imageProviders"` // Sending notification to Slack, Webhook… Notifications Notifications `json:"notifications"` // How the sealed secret should be managed. SealedSecretManagement *SealedSecretManagement `json:"sealedSecretManagement"` - // Optional settings for image watcher. - ImageWatcher PipedImageWatcher `json:"imageWatcher"` // Optional settings for event watcher. EventWatcher PipedEventWatcher `json:"eventWatcher"` } @@ -90,9 +86,6 @@ func (s *PipedSpec) Validate() error { return err } } - if err := s.ImageWatcher.Validate(); err != nil { - return err - } return nil } @@ -398,86 +391,6 @@ type AnalysisProviderStackdriverConfig struct { ServiceAccountFile string `json:"serviceAccountFile"` } -type PipedImageProvider struct { - Name string `json:"name"` - Type model.ImageProviderType `json:"type"` - - DockerHubConfig *ImageProviderDockerHubConfig - GCRConfig *ImageProviderGCRConfig - ECRConfig *ImageProviderECRConfig -} - -type genericPipedImageProvider struct { - Name string `json:"name"` - Type model.ImageProviderType `json:"type"` - - Config json.RawMessage `json:"config"` -} - -func (p *PipedImageProvider) UnmarshalJSON(data []byte) error { - var err error - gp := genericPipedImageProvider{} - if err = json.Unmarshal(data, &gp); err != nil { - return err - } - p.Name = gp.Name - p.Type = gp.Type - - switch p.Type { - case model.ImageProviderTypeDockerHub: - p.DockerHubConfig = &ImageProviderDockerHubConfig{} - if len(gp.Config) > 0 { - err = json.Unmarshal(gp.Config, p.DockerHubConfig) - } - case model.ImageProviderTypeGCR: - p.GCRConfig = &ImageProviderGCRConfig{} - if len(gp.Config) > 0 { - err = json.Unmarshal(gp.Config, p.GCRConfig) - } - case model.ImageProviderTypeECR: - p.ECRConfig = &ImageProviderECRConfig{} - if len(gp.Config) > 0 { - err = json.Unmarshal(gp.Config, p.ECRConfig) - } - default: - err = fmt.Errorf("unsupported image provider type: %s", p.Name) - } - return err -} - -type ImageProviderGCRConfig struct { - // Path to the json file of service account with the required "roles/storage.objectViewer" role. - ServiceAccountFile string `json:"serviceAccountFile"` -} - -type ImageProviderDockerHubConfig struct { - Username string `json:"username"` - PasswordFile string `json:"passwordFile"` -} - -type ImageProviderECRConfig struct { - // The region to send requests to. This parameter is required. - // e.g. "us-west-2" - // A full list of regions is: https://docs.aws.amazon.com/general/latest/gr/rande.html - Region string `json:"region"` - // The AWS account ID associated with the registry that contains the repository - // in which to list images. The "default" registry is assumed by default. - RegistryID string `json:"registryId"` - // Path to the shared credentials file. - // - // Piped attempts to retrieve credentials in the following order: - // 1. from the environment variables. Available environment variables are: - // - AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY - // - AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY - // 2. from the given credentials file. - // 3. from the EC2 Instance Role - CredentialsFile string `json:"credentialsFile"` - // AWS Profile to extract credentials from the shared credentials file. - // If empty, the environment variable "AWS_PROFILE" is used. - // "default" is populated if the environment variable is also not set. - Profile string `json:"profile"` -} - type Notifications struct { // List of notification routes. Routes []NotificationRoute `json:"routes"` @@ -603,45 +516,6 @@ func (p *SealedSecretManagement) UnmarshalJSON(data []byte) error { return err } -type PipedImageWatcher struct { - // Interval to compare the image in the git repository - // and one in the images provider. Default is 5m. - CheckInterval Duration `json:"checkInterval"` - // Settings for each git repository. - GitRepos []PipedImageWatcherGitRepo `json:"gitRepos"` -} - -// Validate checks if: -// - empty repo ids exist -// - duplicated repository settings exist -func (p *PipedImageWatcher) Validate() error { - repos := make(map[string]struct{}, len(p.GitRepos)) - for i := 0; i < len(p.GitRepos); i++ { - if p.GitRepos[i].RepoID == "" { - return fmt.Errorf("repoId is required", p.GitRepos[i].RepoID) - } - if _, ok := repos[p.GitRepos[i].RepoID]; ok { - return fmt.Errorf("duplicated repo id (%s) found in the imageWatcher directive", p.GitRepos[i].RepoID) - } - repos[p.GitRepos[i].RepoID] = struct{}{} - } - return nil -} - -type PipedImageWatcherGitRepo struct { - // Id of the git repository. This must be unique within - // the repos' elements. - RepoID string `json:"repoId"` - // The commit message used to push after updating image. - // Default message is used if not given. - CommitMessage string `json:"commitMessage"` - // The paths to ImageWatcher files to be included. - Includes []string `json:"includes"` - // The paths to ImageWatcher files to be excluded. - // This is prioritized if both includes and this are given. - Excludes []string `json:"excludes"` -} - type PipedEventWatcher struct { // Interval to fetch the latest event and compare it with one defined in EventWatcher config files CheckInterval Duration `json:"checkInterval"` diff --git a/pkg/config/piped_test.go b/pkg/config/piped_test.go index d22088c9d5..3137a49f94 100644 --- a/pkg/config/piped_test.go +++ b/pkg/config/piped_test.go @@ -148,33 +148,6 @@ func TestPipedConfig(t *testing.T) { }, }, }, - ImageProviders: []PipedImageProvider{ - { - Name: "my-dockerhub", - Type: "DOCKER_HUB", - DockerHubConfig: &ImageProviderDockerHubConfig{ - Username: "foo", - PasswordFile: "/etc/piped-secret/dockerhub-pass", - }, - }, - { - Name: "my-gcr", - Type: "GCR", - GCRConfig: &ImageProviderGCRConfig{ - ServiceAccountFile: "/etc/piped-secret/gcr-service-account.json", - }, - }, - { - Name: "my-ecr", - Type: "ECR", - ECRConfig: &ImageProviderECRConfig{ - Region: "us-west-2", - RegistryID: "default", - CredentialsFile: "/etc/piped-secret/aws-credentials", - Profile: "user1", - }, - }, - }, Notifications: Notifications{ Routes: []NotificationRoute{ { @@ -221,16 +194,6 @@ func TestPipedConfig(t *testing.T) { PublicKeyFile: "/etc/piped-secret/sealing-public-key", }, }, - ImageWatcher: PipedImageWatcher{ - CheckInterval: Duration(10 * time.Minute), - GitRepos: []PipedImageWatcherGitRepo{ - { - RepoID: "foo", - CommitMessage: "Update image", - Includes: []string{"imagewatcher-dev.yaml", "imagewatcher-stg.yaml"}, - }, - }, - }, EventWatcher: PipedEventWatcher{ CheckInterval: Duration(10 * time.Minute), GitRepos: []PipedEventWatcherGitRepo{ @@ -258,25 +221,25 @@ func TestPipedConfig(t *testing.T) { } } -func TestPipedImageWatcherValidate(t *testing.T) { +func TestPipedEventWatcherValidate(t *testing.T) { testcases := []struct { name string - imageWatcher PipedImageWatcher + eventWatcher PipedEventWatcher wantErr bool - wantPipedImageWatcher PipedImageWatcher + wantPipedEventWatcher PipedEventWatcher }{ { name: "missing repo id", wantErr: true, - imageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + eventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "", }, }, }, - wantPipedImageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + wantPipedEventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "", }, @@ -286,8 +249,8 @@ func TestPipedImageWatcherValidate(t *testing.T) { { name: "duplicated repo exists", wantErr: true, - imageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + eventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "foo", }, @@ -296,8 +259,8 @@ func TestPipedImageWatcherValidate(t *testing.T) { }, }, }, - wantPipedImageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + wantPipedEventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "foo", }, @@ -310,8 +273,8 @@ func TestPipedImageWatcherValidate(t *testing.T) { { name: "repos are unique", wantErr: false, - imageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + eventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "foo", }, @@ -320,8 +283,8 @@ func TestPipedImageWatcherValidate(t *testing.T) { }, }, }, - wantPipedImageWatcher: PipedImageWatcher{ - GitRepos: []PipedImageWatcherGitRepo{ + wantPipedEventWatcher: PipedEventWatcher{ + GitRepos: []PipedEventWatcherGitRepo{ { RepoID: "foo", }, @@ -334,9 +297,9 @@ func TestPipedImageWatcherValidate(t *testing.T) { } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { - err := tc.imageWatcher.Validate() + err := tc.eventWatcher.Validate() assert.Equal(t, tc.wantErr, err != nil) - assert.Equal(t, tc.wantPipedImageWatcher, tc.imageWatcher) + assert.Equal(t, tc.wantPipedEventWatcher, tc.eventWatcher) }) } } diff --git a/pkg/config/testdata/.pipe/image-watcher.yaml b/pkg/config/testdata/.pipe/image-watcher.yaml deleted file mode 100644 index 7113817616..0000000000 --- a/pkg/config/testdata/.pipe/image-watcher.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: pipecd.dev/v1beta1 -kind: ImageWatcher -spec: - targets: - - image: gcr.io/pipecd/foo - provider: my-gcr - filePath: foo/deployment.yaml - field: $.spec.template.spec.containers[0].image - - image: pipecd/bar - provider: my-dockerhub - filePath: bar/deployment.yaml - field: $.spec.template.spec.containers[0].image diff --git a/pkg/config/testdata/application/k8s-app-use-image-watcher.yaml b/pkg/config/testdata/application/k8s-app-use-image-watcher.yaml deleted file mode 100644 index 7fdabfd64e..0000000000 --- a/pkg/config/testdata/application/k8s-app-use-image-watcher.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: pipecd.dev/v1beta1 -kind: KubernetesApp -spec: - imageWatcher: - targets: - - provider: my-dockerhub - image: pipecd/helloworld - path: - filename: deployment.yaml - field: spec.containers[0].image diff --git a/pkg/config/testdata/piped/piped-config.yaml b/pkg/config/testdata/piped/piped-config.yaml index 07cb7b319e..152f987dd6 100644 --- a/pkg/config/testdata/piped/piped-config.yaml +++ b/pkg/config/testdata/piped/piped-config.yaml @@ -77,24 +77,6 @@ spec: config: serviceAccountFile: /etc/piped-secret/gcp-service-account.json - imageProviders: - - name: my-dockerhub - type: DOCKER_HUB - config: - username: foo - passwordFile: /etc/piped-secret/dockerhub-pass - - name: my-gcr - type: GCR - config: - serviceAccountFile: /etc/piped-secret/gcr-service-account.json - - name: my-ecr - type: ECR - config: - region: us-west-2 - registryId: default - credentialsFile: /etc/piped-secret/aws-credentials - profile: user1 - notifications: routes: - name: dev-slack @@ -132,15 +114,6 @@ spec: # decryptServiceAccountFile: /etc/piped-secret/decrypt-service-account.json # encryptServiceAccountFile: /etc/piped-secret/encrypt-service-account.json - imageWatcher: - checkInterval: 10m - gitRepos: - - repoId: foo - commitMessage: Update image - includes: - - imagewatcher-dev.yaml - - imagewatcher-stg.yaml - eventWatcher: checkInterval: 10m gitRepos: diff --git a/pkg/model/BUILD.bazel b/pkg/model/BUILD.bazel index 47961631a7..144c256417 100644 --- a/pkg/model/BUILD.bazel +++ b/pkg/model/BUILD.bazel @@ -55,8 +55,6 @@ go_library( "environment.go", "event.go", "filestore.go", - "image_name.go", - "imageprovider.go", "model.go", "notificationevent.go", "piped.go", @@ -81,7 +79,6 @@ go_test( "apikey_test.go", "common_test.go", "event_test.go", - "image_name_test.go", "model_test.go", "piped_test.go", "project_test.go", diff --git a/pkg/model/image_name.go b/pkg/model/image_name.go deleted file mode 100644 index 02da26aeac..0000000000 --- a/pkg/model/image_name.go +++ /dev/null @@ -1,62 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package model - -import ( - "fmt" - "path" -) - -// ImageName represents an untagged image. Note that images may have -// the domain omitted (e.g. Docker Hub). If they only have single path element, -// the prefix `library` is implied. -// -// Examples: -// - alpine -// - library/alpine -// - gcr.io/pipecd/helloworld -type ImageName struct { - Domain string - Repo string -} - -func (i ImageName) String() string { - return path.Join(i.Domain, i.Repo) -} - -// Name gives back just repository name without domain. -func (i ImageName) Name() string { - return i.Repo -} - -// ImageRef represents a tagged image. The tag is allowed to be -// empty, though it is in general undefined what that means -// -// Examples: -// - alpine:3.0 -// - library/alpine:3.0 -// - gcr.io/pipecd/helloworld:0.1.0 -type ImageRef struct { - ImageName - Tag string -} - -func (i ImageRef) String() string { - if i.Tag == "" { - return i.ImageName.String() - } - - return fmt.Sprintf("%s:%s", i.ImageName.String(), i.Tag) -} diff --git a/pkg/model/image_name_test.go b/pkg/model/image_name_test.go deleted file mode 100644 index 3f0bf66151..0000000000 --- a/pkg/model/image_name_test.go +++ /dev/null @@ -1,83 +0,0 @@ -package model - -import ( - "testing" - - "github.com/stretchr/testify/assert" -) - -func TestImageNameString(t *testing.T) { - testcases := []struct { - name string - domain string - repo string - want string - }{ - { - name: "empty repo", - want: "", - }, - { - name: "domain omitted", - repo: "repo", - want: "repo", - }, - { - name: "with domain", - domain: "domain", - repo: "repo", - want: "domain/repo", - }, - } - for _, tc := range testcases { - t.Run(tc.name, func(t *testing.T) { - i := ImageName{ - Repo: tc.repo, - Domain: tc.domain, - } - got := i.String() - assert.Equal(t, tc.want, got) - }) - } -} - -func TestImageRefString(t *testing.T) { - testcases := []struct { - name string - imageName ImageName - tag string - want string - }{ - { - name: "empty repo", - want: "", - }, - { - name: "tag omitted", - imageName: ImageName{ - Domain: "domain", - Repo: "repo", - }, - want: "domain/repo", - }, - { - name: "with tag", - imageName: ImageName{ - Domain: "domain", - Repo: "repo", - }, - tag: "tag", - want: "domain/repo:tag", - }, - } - for _, tc := range testcases { - t.Run(tc.name, func(t *testing.T) { - i := ImageRef{ - ImageName: tc.imageName, - Tag: tc.tag, - } - got := i.String() - assert.Equal(t, tc.want, got) - }) - } -} diff --git a/pkg/model/imageprovider.go b/pkg/model/imageprovider.go deleted file mode 100644 index 48a8223c5d..0000000000 --- a/pkg/model/imageprovider.go +++ /dev/null @@ -1,27 +0,0 @@ -// Copyright 2020 The PipeCD Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package model - -type ImageProviderType string - -const ( - ImageProviderTypeDockerHub ImageProviderType = "DOCKER_HUB" - ImageProviderTypeGCR ImageProviderType = "GCR" - ImageProviderTypeECR ImageProviderType = "ECR" -) - -func (t ImageProviderType) String() string { - return string(t) -}