Ensure to watch image from image providers #1117

nakabonne · 2020-11-18T09:02:42Z

What this PR does / why we need it:
Base implementation of #1114.

Sorry for the huge patch. I'd be happy to work on tiny issues respectively after this.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

NONE

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/gcr/gcr.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-11-18T09:11:29Z

Code coverage for golang is 33.01%. This pull request decreases coverage by -0.05%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`57.14%`	`+1.59%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`

pipecd-bot · 2020-11-19T05:41:45Z

The golinter build is completed with FAILURE. The build will be triggered again when you push any other commits. Or you can trigger it manually by /golinter trigger command right now.

You can check the build log from here.

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/model/image_name.go

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-11-19T05:46:59Z

Code coverage for golang is 32.89%. This pull request decreases coverage by -0.05%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`57.14%`	`+1.59%`

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-11-19T09:16:29Z

Code coverage for golang is 32.89%. This pull request decreases coverage by -0.05%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`57.14%`	`+1.59%`

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-11-24T02:04:34Z

Code coverage for golang is 32.90%. This pull request decreases coverage by -0.05%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`57.14%`	`+1.59%`

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-11-26T09:11:22Z

Code coverage for golang is 32.98%. This pull request increases coverage by 0.03%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`85.71%`	`+30.16%`

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

pipecd-bot · 2020-12-01T08:25:52Z

Code coverage for golang is 33.30%. This pull request increases coverage by 0.03%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`0.00%`	`+0.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`85.71%`	`+30.16%`

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/app/piped/imageprovider/provider.go

pkg/app/piped/imagewatcher/watcher.go

pkg/config/image_watcher.go

nghialv · 2020-12-07T03:11:58Z

pkg/app/piped/imageprovider/provider.go

+	"github.com/pipe-cd/pipe/pkg/model"
+)
+
+// Provider acs as a container registry client.


nit: acts

nghialv · 2020-12-07T07:03:27Z

pkg/model/image_name.go

+	Repo   string
+}
+
+func (i ImageName) String() string {


return path.Join(i.Domain, i.Repo)

nghialv · 2020-12-07T07:06:26Z

pkg/model/image_name.go

+	if i.Tag != "" {
+		tag = ":" + i.Tag
+	}
+	return fmt.Sprintf("%s%s", i.ImageName.String(), tag)


nit:

if i.Tag == "" { return i.ImageName.String() } return fmt.Sprintf("%s:%s", i.ImageTag.String(), i.Tag)

nghialv · 2020-12-07T07:08:14Z

pkg/config/piped.go

 	ECRConfig       *ImageProviderECRConfig
 }

 type genericPipedImageProvider struct {


line 384: Dockerhub -> DockerHub

nghialv · 2020-12-07T07:09:04Z

pkg/app/piped/imageprovider/provider.go

+	switch cfg.Type {
+	case model.ImageProviderTypeGCR:
+		return gcr.NewProvider(cfg.Name, cfg.GCRConfig, doChallenge, logger)
+	case model.ImageProviderTypeDockerhub:


Dockerhub -> DockerHub

nghialv · 2020-12-07T07:27:04Z

pkg/config/piped.go

-	Type model.ImageProviderType
+	Name string                  `json:"name"`
+	Type model.ImageProviderType `json:"type"`
+	// Default is five minute.


nit: 5m (to be same with the specifying in the config file)

nghialv · 2020-12-07T07:51:22Z

pkg/app/piped/imageprovider/gcr/gcr.go

+	if err != nil {
+		return nil, err
+	}
+	_, err = repository.Tags(ctx).All(ctx)


As they described:

For repositories with a large number of tags, this response may be quite large. If such a response is expected, one should use the pagination.

https://docs.docker.com/registry/spec/api/#listing-image-tags

Sorry for the lack of description, the implementation of GCR is still WIP, and we decided to prioritize ECR over GCR. So will just leave TODO comments.

nghialv · 2020-12-07T08:07:17Z

pkg/app/piped/imagewatcher/watcher.go

+
+		go w.run(ctx, p, cfg.PullInterval.Duration())
+	}
+	return nil


This is exited immediately after starting goroutines for all image providers.
Please ensure that all of its child tasks must be stopped before exiting.

nghialv · 2020-12-07T08:14:15Z

pkg/app/piped/imagewatcher/watcher.go

+			}
+			includes = append(includes, target.Includes...)
+			excludes = append(excludes, target.Excludes...)
+		}


nit: Instead of finding and saving these include/exclude list, just finding and marking the target.

Okay, let me address it when I work on this 🙏 :
https://github.com/pipe-cd/pipe/blob/1c4d30e9b310b4e180af234efcffe9e142ba4d7d/pkg/config/image_watcher.go#L31-L32

nghialv · 2020-12-07T08:22:45Z

pkg/app/piped/imagewatcher/watcher.go

+			return true
+		}
+		branch := repo.GetClonedBranch()
+		if err := repo.Pull(ctx, branch); err != nil {


git.Repo is a pointer and pointing to the same disk location of git data.
This Pull can be called from multiple goroutines and may cause a data race.

nghialv · 2020-12-07T08:24:29Z

@nakabonne Reviewing is not finished yet, but please check the comments.

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/model/imageprovider.go

nakabonne · 2020-12-07T09:37:00Z

@nghialv Fixed. Could you take a look?

pipecd-bot · 2020-12-07T09:41:07Z

Code coverage for golang is 33.38%. This pull request increases coverage by 0.10%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`100.00%`	`+100.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`100.00%`	`+100.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`85.71%`	`+30.16%`

nghialv · 2020-12-07T13:22:18Z

pkg/model/imageprovider.go


 const (
-	ImageProviderTypeDockerhub ImageProviderType = "DOCKERHUB"
+	ImageProviderTypeDockerHub ImageProviderType = "DOCKERHUB"


DOCKER_HUB

nghialv · 2020-12-08T04:44:49Z

pkg/app/piped/imagewatcher/watcher.go

+	mu        sync.Mutex
+
+	// Indexed by repo id.
+	gitRepos sync.Map


Not sure why do we need to use a sync map instead of a normal map?

Oops, you're right. A map can be read when it's being read. Gonna fix them.

pipecd-bot · 2020-12-08T09:10:17Z

The following ISSUES will be created once got merged. If you want me to skip creating the issue, you can use /todo skip command.

Details

1. Stop listing all tags

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imageprovider/gcr/gcr.go#L90-L93

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

2. Give back latest image from GCR

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imageprovider/gcr/gcr.go#L95-L98

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

3. Use credentials for GCR configured by user

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imageprovider/gcr/gcr.go#L100-L103

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

4. Handle the no challenge case

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imageprovider/provider.go#L68-L71

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

5. Control not to reach the rate limit

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imagewatcher/watcher.go#L173-L176

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

6. Compares between image repos in the image registry and image repos in git

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imagewatcher/watcher.go#L178-L181

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

7. Make it possible to push outdated images to Git

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/app/piped/imagewatcher/watcher.go#L186-L188

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

8. Load image watcher config

https://github.com/pipe-cd/pipe/blob/dc28c17cdf7b385830ef789b0e00d907b47f5196/pkg/config/image_watcher.go#L31-L34

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

pipecd-bot

Some issues were detected while linting go source files in your changes.

pkg/model/imageprovider.go

nakabonne · 2020-12-08T09:14:33Z

@nghialv I really appreciate your polite review. Please take a look again 😄

pipecd-bot · 2020-12-08T09:16:08Z

Code coverage for golang is 33.38%. This pull request increases coverage by 0.10%.

File	Function	Base	Head	Diff
pkg/config/image_watcher.go	LoadImageWatcher	`--`	`0.00%`	`+0.00%`
pkg/config/image_watcher.go	ImageWatcherSpec.Validate	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageName.String	`--`	`100.00%`	`+100.00%`
pkg/model/image_name.go	ImageName.Name	`--`	`0.00%`	`+0.00%`
pkg/model/image_name.go	ImageRef.String	`--`	`100.00%`	`+100.00%`
pkg/config/config.go	Config.init	`66.67%`	`60.87%`	`-5.80%`
pkg/config/piped.go	PipedImageProvider.UnmarshalJSON	`55.56%`	`85.71%`	`+30.16%`

nghialv · 2020-12-08T13:12:20Z

Nice.
/approve

pipecd-bot · 2020-12-08T13:12:24Z

This pull request is APPROVED by nghialv.

Approvers can cancel the approval by writing /approve cancel in a comment. Any additional commits also will change this pull request to be not-approved.

nakabonne · 2020-12-08T13:13:15Z

Thanks 😃

pipecd-bot added release-note-none do-not-merge/wip labels Nov 18, 2020

pipecd-bot requested review from gkuga and stormcat24 November 18, 2020 09:02

pipecd-bot added area/piped size/L labels Nov 18, 2020

pipecd-bot reviewed Nov 18, 2020

View reviewed changes

pkg/app/piped/imageprovider/gcr/gcr.go Outdated Show resolved Hide resolved

pkg/app/piped/imageprovider/gcr/gcr.go Outdated Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Outdated Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from 15c2474 to c4ef830 Compare November 19, 2020 05:41

pipecd-bot added size/XL area/build and removed size/L labels Nov 19, 2020

nakabonne force-pushed the image-watcher branch from c4ef830 to 8995bb2 Compare November 19, 2020 05:41

pipecd-bot reviewed Nov 19, 2020

View reviewed changes

pkg/model/image_name.go Outdated Show resolved Hide resolved

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from 8995bb2 to 177f903 Compare November 19, 2020 09:11

pipecd-bot reviewed Nov 19, 2020

View reviewed changes

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from 177f903 to bff5808 Compare November 24, 2020 01:58

pipecd-bot reviewed Nov 24, 2020

View reviewed changes

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from bff5808 to fc2edcd Compare November 26, 2020 09:05

pipecd-bot reviewed Nov 26, 2020

View reviewed changes

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from fc2edcd to 49576fe Compare December 1, 2020 08:19

pipecd-bot reviewed Dec 1, 2020

View reviewed changes

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nakabonne force-pushed the image-watcher branch from 49576fe to 23f7fb3 Compare December 2, 2020 09:01

pipecd-bot reviewed Dec 2, 2020

View reviewed changes

pkg/app/piped/imageprovider/provider.go Show resolved Hide resolved

pkg/app/piped/imagewatcher/watcher.go Show resolved Hide resolved

pkg/config/image_watcher.go Outdated Show resolved Hide resolved

nghialv reviewed Dec 7, 2020

View reviewed changes

nakabonne added 3 commits December 7, 2020 17:57

Fix image name string

2cc2c9f

Fix nits

44f960b

Fix

02c9d9b

pipecd-bot reviewed Dec 7, 2020

View reviewed changes

pkg/model/imageprovider.go Outdated Show resolved Hide resolved

nghialv reviewed Dec 8, 2020

View reviewed changes

Use normal map instead of sync.Map

dc28c17

pipecd-bot reviewed Dec 8, 2020

View reviewed changes

pkg/model/imageprovider.go Show resolved Hide resolved

nghialv changed the title ~~Ensure to call image providers periodically~~ Ensure to watch image from image providers Dec 8, 2020

pipecd-bot added the approved label Dec 8, 2020

pipecd-bot merged commit 219102c into master Dec 8, 2020

pipecd-bot deleted the image-watcher branch December 8, 2020 13:19

pipecd-bot mentioned this pull request Dec 16, 2020

Add doc for pipectl #1266

Merged

Ensure to watch image from image providers #1117

Ensure to watch image from image providers #1117

Uh oh!

Conversation

nakabonne commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Nov 18, 2020

Uh oh!

pipecd-bot commented Nov 19, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Nov 19, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Nov 19, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Nov 24, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Nov 26, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pipecd-bot commented Dec 1, 2020

Uh oh!

pipecd-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nakabonne commented Nov 18, 2020 •

edited

Loading

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.

This was created by todo plugin since "TODO:" was found in `dc28c17` when #1117 was merged. cc: @nakabonne.