feat: add syncset controller, feat: syncset readiness by acpana · Pull Request #3030 · open-policy-agent/gatekeeper

acpana · 2023-10-03T20:59:05Z

in support of the sync set proposal

Adds a syncset_controller and necessary changes for tracking readiness from syncsets and config resources
Introduces a reconciler for data expectations
Implements "TryCancel" for data expectations
Improves the CacheManager to distinguish the watch errors for the gvks it's given
Adds the SyncSet CRD to the manifest.

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe

Added comments, have not reviewed tests.

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

plus naming, style, etc Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

codecov-commenter · 2023-10-06T07:49:18Z

Codecov Report

❌ Patch coverage is 70.42254% with 126 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.78%. Comparing base (aed792f) to head (430deea).
⚠️ Report is 685 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/controller/syncset/syncset_controller.go	46.47%	31 Missing and 7 partials ⚠️
pkg/readiness/ready_tracker.go	74.46%	30 Missing and 6 partials ⚠️
pkg/cachemanager/cachemanager.go	76.62%	15 Missing and 3 partials ⚠️
pkg/readiness/pruner/pruner.go	65.51%	9 Missing and 1 partial ⚠️
pkg/cachemanager/aggregator/aggregator.go	50.00%	6 Missing and 1 partial ⚠️
pkg/watch/errorlist.go	81.57%	7 Missing ⚠️
pkg/watch/set.go	0.00%	5 Missing ⚠️
pkg/watch/manager.go	25.00%	2 Missing and 1 partial ⚠️
pkg/controller/config/config_controller.go	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3030      +/-   ##
==========================================
+ Coverage   53.21%   53.78%   +0.56%     
==========================================
  Files         134      136       +2     
  Lines       11917    12198     +281     
==========================================
+ Hits         6342     6561     +219     
- Misses       5080     5136      +56     
- Partials      495      501       +6

Flag	Coverage Δ
unittests	`53.78% <70.42%> (+0.56%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

- rework interpretErr to return failing gvks - implement TryCancel for trackerMap Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe

Getting closer!

Have not reviewed tests.

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

ritazh · 2023-10-11T16:25:00Z

@acpana can you please update #2393 to include what this PR covers and what still remains in order to close that issue out?

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

julianKatz

Great work!! Way to stick with it.

passing the baton to Max to close things out on Google approval

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

anlandu · 2023-11-11T00:51:43Z

Looks great!!

maxsmythe

Once the bug in cache manager is addressed and the aggregator test has assertions, LGTM

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe

Two minor fixes, then LGTM

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe

LGTM

ritazh

minor comments

Co-authored-by: Rita Zhang <rita.z.zhang@gmail.com> Signed-off-by: alex <8968914+acpana@users.noreply.github.com>

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

ritazh

LGTM

Thanks for getting this feature cross the finishing line! 🎉

acpana added 4 commits October 3, 2023 20:21

add syncset controller

e34eed0

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

add syncset readiness, observe config

3d43814

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

reconcile data expectations

0dc69bd

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

enable syncset crd

46dfde2

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe requested changes Oct 5, 2023

View reviewed changes

acpana added 3 commits October 5, 2023 01:23

interpret registrar err

686af5d

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

review: move EM, rename to EP

7e94cdd

plus naming, style, etc Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

review: remove waitgroup

31979ac

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe October 6, 2023 18:08

smoll fix for test

b290867

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana mentioned this pull request Oct 7, 2023

refactor: remove finalizer code #3043

Closed

acpana added 3 commits October 9, 2023 15:43

Merge branch 'master' into acpana/syncsets-rt-r

49d652e

use logging constant

2192d65

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

add TryCancelData

592e243

- rework interpretErr to return failing gvks - implement TryCancel for trackerMap Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana mentioned this pull request Oct 9, 2023

refactor: more debug logs in RT #3056

Closed

fix lock

8da842c

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana marked this pull request as ready for review October 9, 2023 23:46

maxsmythe reviewed Oct 10, 2023

View reviewed changes

acpana added 3 commits October 11, 2023 02:41

review: rm comments, mgr starts pruner

50f7abd

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

review: move interface, IsUniversal

7296196

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

review: explicitly remove source

6a260a3

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe October 11, 2023 06:32

ritazh added this to the v3.14.0 milestone Oct 11, 2023

acpana mentioned this pull request Oct 12, 2023

Add support for metadata.gatekeeper.sh/requires-sync-data #2393

Open

Merge branch 'master' into acpana/syncsets-rt-r

a1e153f

julianKatz reviewed Oct 12, 2023

View reviewed changes

acpana added 2 commits October 13, 2023 20:59

limit name to 63 char

689b252

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

review: comments, naming, HasBalidationOperation

4708b4d

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe reviewed Nov 6, 2023

View reviewed changes

Comment thread pkg/watch/errorlist.go Outdated

Comment thread pkg/watch/errorlist.go Outdated

acpana added 2 commits November 7, 2023 02:06

check err nullability, naming

eeba2bf

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

use remove set instead of forceWipe

034e826

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe November 7, 2023 22:10

maxsmythe reviewed Nov 8, 2023

View reviewed changes

Comment thread pkg/cachemanager/cachemanager.go Outdated

set instead of book for dangling watches

854306e

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe November 9, 2023 19:15

maxsmythe reviewed Nov 9, 2023

View reviewed changes

Comment thread pkg/cachemanager/cachemanager.go Outdated

general error watch dangling

d7bb9b5

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe November 10, 2023 00:19

julianKatz approved these changes Nov 10, 2023

View reviewed changes

review feedback

7faecbc

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe requested changes Nov 14, 2023

View reviewed changes

Comment thread pkg/cachemanager/aggregator/aggregator_test.go Outdated

Comment thread pkg/cachemanager/cachemanager.go Outdated

Comment thread pkg/readiness/pruner/pruner_test.go

Comment thread pkg/readiness/ready_tracker_unit_test.go Outdated

acpana added 2 commits November 15, 2023 23:53

refactor: use tt

03565fa

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

fix: general err dangling watches

3766d66

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

acpana requested a review from maxsmythe November 16, 2023 00:01

maxsmythe requested changes Nov 16, 2023

View reviewed changes

Comment thread pkg/cachemanager/aggregator/aggregator_test.go

Comment thread pkg/cachemanager/cachemanager.go Outdated

rf: add set directly

352ae72

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

maxsmythe approved these changes Nov 16, 2023

View reviewed changes

Merge branch 'master' into acpana/syncsets-rt-r

a216503

acpana requested review from ritazh and sozercan November 17, 2023 00:30

ritazh reviewed Nov 22, 2023

View reviewed changes

acpana and others added 3 commits November 22, 2023 11:59

Apply suggestions from code review

a280fb7

Co-authored-by: Rita Zhang <rita.z.zhang@gmail.com> Signed-off-by: alex <8968914+acpana@users.noreply.github.com>

review suggestions

c87e934

Signed-off-by: Alex Pana <8968914+acpana@users.noreply.github.com>

Merge branch 'master' into acpana/syncsets-rt-r

430deea

acpana requested a review from ritazh November 22, 2023 20:11

ritazh approved these changes Nov 22, 2023

View reviewed changes

acpana merged commit 6276cb2 into open-policy-agent:master Nov 23, 2023

Conversation

acpana commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ritazh commented Oct 11, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julianKatz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anlandu commented Nov 11, 2023

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

ritazh left a comment

acpana commented Oct 3, 2023 •

edited

Loading

codecov-commenter commented Oct 6, 2023 •

edited

Loading