feat: add generate operation and wait for VAPB generation by JaydipGabani · Pull Request #3573 · open-policy-agent/gatekeeper

JaydipGabani · 2024-10-05T00:57:42Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #3659
Related #3501

Special notes for your reviewer:

codecov-commenter · 2024-10-05T02:17:50Z

Codecov Report

Attention: Patch coverage is 27.94118% with 196 lines in your changes missing coverage. Please review.

Project coverage is 47.64%. Comparing base (3350319) to head (a6ac8fd).
Report is 392 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/controller/constraint/constraint_controller.go	0.00%	118 Missing ⚠️
...onstrainttemplate/constrainttemplate_controller.go	49.35%	60 Missing and 18 partials ⚠️

❗ There is a different number of reports uploaded between BASE (3350319) and HEAD (a6ac8fd). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (3350319) HEAD (a6ac8fd)

unittests 2 1

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3573      +/-   ##
==========================================
- Coverage   54.49%   47.64%   -6.85%     
==========================================
  Files         134      236     +102     
  Lines       12329    19786    +7457     
==========================================
+ Hits         6719     9428    +2709     
- Misses       5116     9476    +4360     
- Partials      494      882     +388

Flag	Coverage Δ
unittests	`47.64% <27.94%> (-6.85%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

maxsmythe

Biggest concern is the timestamp stuff.

maxsmythe · 2024-10-15T01:44:38Z

-	DefaultGenerateVAP  = flag.Bool("default-create-vap-for-templates", false, "Create VAP resource for template containing VAP-style CEL source. Allowed values are false: do not create Validating Admission Policy unless generateVAP: true is set on constraint template explicitly, true: create Validating Admission Policy unless generateVAP: false is set on constraint template explicitly.")
+	log                      = logf.Log.V(logging.DebugLevel).WithName("controller").WithValues(logging.Process, "constraint_controller")
+	discoveryErr             *apiutil.ErrResourceDiscoveryFailed
+	DefaultWaitForGeneration = flag.Int("default-wait-for-generation", 30, "Wait to generate ValidatingAdmissionPolicyBinding after the constraint is created. Defaults to 30 seconds.")


Probably should not include the hardcoded time in the error text -- prone to bitrot.

maxsmythe · 2024-10-15T01:51:10Z

 					}
-					currentVapBinding = nil
+
+					if currentVapBinding == nil && instance.GetCreationTimestamp().Add(time.Duration(*DefaultWaitForGeneration)).Before(time.Now()) {


We should not rely on the creation timestamp... that leaves us vulnerable to clock skew since that is set by an unknown machine.

Also, as discussed in the OSS mtg, the time delay should be based off of the constraint template, specifically wait time seconds after the CRD is created from the template. There is no need to wait due to the constraint.

maxsmythe · 2024-10-15T02:14:54Z

-			if generateVAPB && groupVersion != nil {
-				currentVapBinding, err := vapBindingForVersion(*groupVersion)
+			if operations.IsAssigned(operations.Generate) {
+				err := util.ValidateEnforcementAction(enforcementAction, instance.Object)


VAP generation should probably be in its own function -- the reconcile function is unwieldy at this point.

Also, I'm not sure we want to only validate enforcement action when generate is enabled.

refactored the code

maxsmythe · 2024-10-15T02:15:21Z

-			return reconcile.Result{}, err
+
+	if operations.IsAssigned(operations.Generate) {
+		isVapAPIEnabled := false


We should probably move generation lifecycle stuff to its own function.

ritazh · 2024-10-15T04:20:05Z

 \n        - --exempt-namespace=${GATEKEEPER_NAMESPACE}\
 \n        - --operation=webhook\
 \n        - --operation=mutation-webhook\
+\n        - --operation=generate\


this is added to controller-manager instead of audit. was this intentional?

This was supposed to be added to audit pod. Fixed it.

ritazh · 2024-10-23T04:58:23Z

 | enableK8sNativeValidation                                  | Enable the K8s Native Validating driver to allow constraint templates to use rules written in VAP-style CEL (beta feature)                                                                                                                                                                                                                 | `true`                                                                                                                                                               |
 | defaultCreateVAPForTemplates                                      | (alpha) Create VAP resource for template containing VAP-style CEL source. Allowed values are false: do not create Validating Admission Policy unless generateVAP: true is set on constraint template explicitly, true: create Validating Admission Policy unless generateVAP: false is set on constraint template explicitly.                                                                                                                                                                                                                  | `false`                                                                                                                                                               |
 | defaultCreateVAPBindingForConstraints                             | (alpha) Create VAPBinding resource for constraint of the template containing VAP-style CEL source. Allowed values are false: do not create Validating Admission Policy Binding, true: create Validating Admission Policy Binding.                                                                                                                                                                                                                  | `false`                                                                                                                                                               |
+| defaultWaitForVAPBGeneration                             | (alpha) Wait to generate ValidatingAdmissionPolicyBinding after the constraint CRD is created.                                                                                                                                                                                                                 | `30`                                                                                                                                                               |


Suggested change

| defaultWaitForVAPBGeneration | (alpha) Wait to generate ValidatingAdmissionPolicyBinding after the constraint CRD is created. | `30` |

| defaultWaitForVAPBGeneration | (alpha) Wait time in seconds before generating a ValidatingAdmissionPolicyBinding after a constraint CRD is created. | `30` |

ritazh · 2024-10-23T04:59:58Z

-	logger       = log.Log.V(logging.DebugLevel).WithName("controller").WithValues("kind", "ConstraintTemplate", logging.Process, "constraint_template_controller")
-	discoveryErr *apiutil.ErrResourceDiscoveryFailed
+	logger                       = log.Log.V(logging.DebugLevel).WithName("controller").WithValues("kind", "ConstraintTemplate", logging.Process, "constraint_template_controller")
+	defaultWaitForVAPBGeneration = flag.Int("default-wait-for-vapb-generation", 30, "(alpha) Wait to generate ValidatingAdmissionPolicyBinding after the constraint CRD is created.")


ritazh · 2024-10-29T01:26:54Z


+func (r *ReconcileConstraint) generateVAPB(ctx context.Context, enforcementAction util.EnforcementAction, instance *unstructured.Unstructured, status *constraintstatusv1beta1.ConstraintPodStatus) (time.Duration, error) {
+	ret := time.Duration(0)
+	if !operations.IsAssigned(operations.Generate) {


if none of the controllers has the generate operation, how do we report this issue to the user?

I have added log statement for now. Another option is to figure the same out by looking at all CTStatus resource in CTstatus controller - https://github.com/open-policy-agent/gatekeeper/blob/master/pkg/controller/constrainttemplatestatus/constrainttemplatestatus_controller.go#L185. However we might need one field under CT Status to post the error.

ritazh · 2024-10-29T01:37:24Z

+		if err != nil {
+			return err
+		}
+		if t.Before(currentTime.Add(time.Duration(*defaultWaitForVAPBGeneration*2) * time.Second)) {


why is this * 2? can you add a comment for this?

I modified this to remove *2, added the comment as well.

ritazh · 2024-10-29T01:40:51Z

+
+	// waiting for sometime before generating vapbinding, gives api-server time to cache CRDs
+	timestamp := ct.Annotations[BlockVAPBGenerationUntilAnnotation]
+	if timestamp != "" {


what if it's not populated

Fixed this behavior.

ritazh · 2024-10-29T01:43:56Z

+func (r *ReconcileConstraintTemplate) updateTemplateWithBlockVAPBGenerationAnnotations(ctx context.Context, ct *v1beta1.ConstraintTemplate) error {
+	currentTime := time.Now()
+	switch {
+	case ct.Annotations == nil || ct.Annotations[constraint.BlockVAPBGenerationUntilAnnotation] == "":


does this need to be a switch statement or can a if clause work better since some of the code below seems to be repeated.

Updated the code to if stmt

ritazh · 2024-10-29T01:48:02Z

+			// currentTime := time.Now()
+			// if currentTime.Before(blockTime) {
+			// 	t.Fatal("VAPBinding should not be created before the timestamp", currentTime, blockTime)
+			// }


why is this commented out?

removed the commented block. Updated the tests as well.

ritazh · 2024-10-30T00:30:37Z

+			return err
+		}
+		// if wait time is within the time window to generate vap binding, do not update the annotation
+		if t.Before(currentTime.Add(time.Duration(*constraint.DefaultWaitForVAPBGeneration) * time.Second)) {


if annotation is already populated, then can we always skip an update? what's the use case for when the annotation needs to be updated with a new time?

This ensures that VAPB is generated if a faulty actor has set the timestamp in the future manually—beyond the allowed delay configured by the flag.

thanks for the context and it would be useful to add that as a comment.

ritazh · 2024-10-30T00:43:33Z

+			default:
+				// reconcile after default wait time for vapb generation if annotation is not set
+				if ct.Annotations == nil || ct.Annotations[BlockVAPBGenerationUntilAnnotation] == "" {
+					return time.Duration(*DefaultWaitForVAPBGeneration) * time.Second, r.reportErrorOnConstraintStatus(ctx, status, errors.New("annotation to wait for ValidatingAdmissionPolicyBinding generation not found"), "could not find annotation to wait for ValidatingAdmissionPolicyBinding generation")


this returned time duration is ignore when an error is returned. we should be consistent to always rely on the annotation for the timestamp instead of returning a duration here.

The purpose here was to requeue after user defined delay in case wanted annotations are not found (wait for annotation to be set by CT constroller if annotation was not present for any reason). I updated the code to reflect the same.

No need to have a user-defined delay when annotation is missing, that should be handled by controller code (user sets time delay after CRD gets created, not internal controller mechanics)

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe · 2024-11-05T01:01:39Z

-						return reconcile.Result{}, r.reportErrorOnConstraintStatus(ctx, status, err, fmt.Sprintf("could not delete ValidatingAdmissionPolicyBinding: %s", vapBindingName))
-					}
-				}
+			if requeueAfter != time.Duration(0) {


because we are requeueing before cacheConstraint is called (which adds the constraint to the constraint framework), the constraint will not be enforced at all until the VAP objects are created. These seems unnecessary and will hurt the performance of existing G8r uses.

maxsmythe · 2024-11-05T01:04:28Z

-						return reconcile.Result{}, r.reportErrorOnConstraintStatus(ctx, status, err, fmt.Sprintf("could not update ValidatingAdmissionPolicyBinding: %s", vapBindingName))
-					}
-				}
+			requeueAfter, err := r.generateVAPB(ctx, enforcementAction, instance, status)


All of the VAP generation logic is gated on reflect.DeepEqual() of the constraint as-cached by the constraint framework.

I don't think that is the right logic gate for VAP-gen logic -- it should be gated on whether the extant VAP objects have drifted from the to-be-generated VAP objects. Otherwise users could modify the generated VAP objects and G8r would not re-align them until the next time the constraint is touched.

It looks like we gate actual writes on VAP binding having a diff --- could just always call generateVAP() code. Maybe we could avoid unnecessary logic execution and therefore improve performance by refactoring more, but always calling VAP gen should be sufficient.

Bonus points for calling it after adding the constraint to the constraint framework -- that avoids blocking all enforcement on VAP succeeding as mentioned elsewhere.

updated to call generateVAP out side of reflect.DeepEqual() gate. Also calling generateVAP after constraint is marked enforced.

maxsmythe · 2024-11-05T01:23:37Z

+			hasVAP, err := ShouldGenerateVAP(unversionedCT)
+			switch {
+			case errors.Is(err, celSchema.ErrCodeNotDefined):
+				generateVAPB = false


Should this error be swallowed? Or should we tell the user that the constraint template does not support VAP?

This will mean when we switch the default to always generate VAP and VAPB, we will be throwing error for existing users using constraint template with Rego. I can also see the other use-case, where we would want to notify users if the intent is to use VAP but there is no CEL in the CT.

@ritazh @sozercan Any thoughts on this?

Did we not switch to a model where users must use the VAP enforcement point to use VAP? I remember having that discussion. Part of the reason for requiring explicit user intent was to avoid questions like this.

In any case, I think an error or "warning"-type error makes sense. Users can use scoped EA to disable it.

Contrast that with users getting no notice that a Rego-sourced CT is incompatible with VAP.

IMO we should check if engine is K8sNativeValidation, then log and report the error. I recall this error was swallowed in 3.17.1 because 3.17.0 was reporting this error for existing rego CTs.

I do not recall having a discussion where user must use the VAP enforcement point to use VAP. I thought eventually we wanted to enable VAP enforcement point by default in-line with other enforcement points.

Reporting error only when VAP enforcement point explicitly mentioned in constraint through scopedEnforcementAction and template does not satisfy conditions to create VAP could work. This will not affect current rego users. And when we switch to VAP by default, the guidance for constraint authors would be:

If VAP is available to use, GK will use VAP with all constraint. Otherwise GK will use webhook for enforcement. No errors reported on CT about not generating VAP.

If VAP is not available to use and user wants to use VAP with scopedEnforcementPoints, GK reports error on CT and enforces through webhook if included as enforcement point.

Does this make sense?^^

As per the community discussion, I will follow up with instroducing field on constraint status for enforcementPoint status under bypod.

maxsmythe · 2024-11-05T01:27:40Z

+			default:
+				// reconcile after default wait time for vapb generation if annotation is not set
+				if ct.Annotations == nil || ct.Annotations[BlockVAPBGenerationUntilAnnotation] == "" {
+					return time.Duration(*DefaultWaitForVAPBGeneration) * time.Second, r.reportErrorOnConstraintStatus(ctx, status, errors.New("annotation to wait for ValidatingAdmissionPolicyBinding generation not found"), "could not find annotation to wait for ValidatingAdmissionPolicyBinding generation")


No need to have a user-defined delay when annotation is missing, that should be handled by controller code (user sets time delay after CRD gets created, not internal controller mechanics)

maxsmythe · 2024-11-05T01:31:05Z

+		vapBindingName := fmt.Sprintf("gatekeeper-%s", instance.GetName())
+		log.Info("check if vapbinding exists", "vapBindingName", vapBindingName)
+		if err := r.reader.Get(ctx, types.NamespacedName{Name: vapBindingName}, currentVapBinding); err != nil {
+			if !apierrors.IsNotFound(err) && !errors.As(err, &discoveryErr) && !meta.IsNoMatchError(err) {


Discovery/nomatch errors should not be a concern for bindings (which are a built-in type)

Updated to remove the error matching.

maxsmythe · 2024-11-05T01:37:36Z

-						return reconcile.Result{}, r.reportErrorOnConstraintStatus(ctx, status, err, fmt.Sprintf("could not update ValidatingAdmissionPolicyBinding: %s", vapBindingName))
-					}
-				}
+			requeueAfter, err := r.generateVAPB(ctx, enforcementAction, instance, status)


It looks like we gate actual writes on VAP binding having a diff --- could just always call generateVAP() code. Maybe we could avoid unnecessary logic execution and therefore improve performance by refactoring more, but always calling VAP gen should be sufficient.

Bonus points for calling it after adding the constraint to the constraint framework -- that avoids blocking all enforcement on VAP succeeding as mentioned elsewhere.

maxsmythe · 2024-11-05T01:43:01Z

+			return err
+		}
+		// if wait time is within the time window to generate vap binding, do not update the annotation
+		// otherwise update the annotation with the current time + wait time. This protects against manual updates on annotations with a timestamp that prevents binding from getting generated.


I think, more accurately, this prevents clock skew from preventing generation on task reschedule.

This design really doesn't protect against malicious users (RBAC for templates does that)

maxsmythe · 2024-11-05T01:44:26Z

+		return nil
+	}
+
+	err := r.updateTemplateWithBlockVAPBGenerationAnnotations(ctx, ct)


Should probably note that we are purposefully making a second call to the API server here in order to make sure the timestamp is post-CRD creation.

Otherwise the question I have is "why make two requests?"

Added a comment here.

maxsmythe · 2024-11-05T01:45:47Z

+	if ct.Annotations == nil {
+		ct.Annotations = make(map[string]string)
+	}
+	ct.Annotations[constraint.BlockVAPBGenerationUntilAnnotation] = currentTime.Add(time.Duration(*constraint.DefaultWaitForVAPBGeneration) * time.Second).Format(time.RFC3339)


We should also have an annotation that is non-clock-dependent, that lets us know when generation is always allowed, otherwise we run the risk of sporadic delays when the pod reschedules.

Likely a significant change, probably not a blocker for beta (just a known risk)

Once the annotations are set after CRD is created, pod reschedules won't cause the annotations to be set again. In the event of pod getting reschedule after CRD is created and before the annotations are set, annotations will get set on the restart and then it won't get updated again.

Since we only update annotations if it is not set or it is in future outside of the window defined by users, I am failing to see the need of non-clock-dependent annotation. Can you describe an example where we run the risk of sporadic delays?

Remember the discussion about clock skew. Pod gets rescheduled to a node that is a day behind -> sporadic 30-second delays generating bindings, even if the CRD has been extant for hours.

Yip, this slipped my mind. Thanks for the context. Let's follow up with this fix. I will create an issue.

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe · 2024-11-06T00:54:34Z

+			default:
+				// reconcile for vapb generation if annotation is not set
+				if ct.Annotations == nil || ct.Annotations[BlockVAPBGenerationUntilAnnotation] == "" {
+					return time.Duration(1) * time.Second, r.reportErrorOnConstraintStatus(ctx, status, errors.New("annotation to wait for ValidatingAdmissionPolicyBinding generation not found"), "could not find annotation to wait for ValidatingAdmissionPolicyBinding generation")


Per previous comment, this should return a duration of zero... let the controller decide what the retry delay is.

ritazh · 2024-11-06T01:24:49Z

+			return err
+		}
+		if ct.Annotations != nil {
+			ct.Annotations[constraint.BlockVAPBGenerationUntilAnnotation] = ""


Do we need to reset this annotation when the CRD needs an update? the CRD should already been created and known to the apiserver right? and therefore, no need to wait for the vapb creation/updates?

Yeah, you are correct. I initially thought that we need to wait because CRD would be different and api-server needs time to cache it. But it really doesn't matter, because all VAP cares about is kind. I will update the PR.

Updated the PR.

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe

LGTM with 2 nits, sorry for delay.

maxsmythe · 2024-11-07T22:12:16Z

+	if generateVAPB && groupVersion != nil {
+		currentVapBinding, err := vapBindingForVersion(*groupVersion)
+		if err != nil {
+			return ret, r.reportErrorOnConstraintStatus(ctx, status, err, "could not get ValidatingAdmissionPolicyBinding API version")


nit: ret is effectively time.Duration(0) (it never gets overriden) for readability, probably best to rename ret something like noDelay or similar.

maxsmythe · 2024-11-07T22:15:19Z

+		return nil
+	}
+
+	// Update templates to make sure that the VAPB generation block until timestamp set post CRD creation


nit: "We add the annotation as a follow-on update to be sure the timestamp is set relative to a time after the CRD is successfully created. Creating the CRD with a delay timestamp already set would not account for request latency"

would be more clear (feel free to update wording). Current comment is more descriptive of the code than explanatory as to why the code is the way it is.

…setting annotation on CT Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe

LGTM

grosser · 2024-12-20T17:00:16Z

can we get some docs on what this generate operation does ?
... and a PR description would also be nice

JaydipGabani changed the title ~~[WIP] chore: adding generate operation~~ chore: adding generate operation Oct 7, 2024

JaydipGabani marked this pull request as ready for review October 7, 2024 22:36

JaydipGabani requested a review from a team as a code owner October 7, 2024 22:36

maxsmythe requested changes Oct 15, 2024

View reviewed changes

ritazh reviewed Oct 15, 2024

View reviewed changes

JaydipGabani force-pushed the operation-generate branch from a0fd55a to a0d015e Compare October 22, 2024 03:06

JaydipGabani requested review from maxsmythe and ritazh October 22, 2024 03:14

ritazh requested changes Oct 29, 2024

View reviewed changes

JaydipGabani requested a review from ritazh October 30, 2024 00:19

ritazh reviewed Oct 30, 2024

View reviewed changes

ritazh added this to the v3.18.0 milestone Oct 30, 2024

JaydipGabani requested a review from ritazh October 30, 2024 01:09

JaydipGabani mentioned this pull request Oct 30, 2024

Update default for vap generation #3501

Closed

JaydipGabani changed the title ~~chore: adding generate operation~~ feat: adding generate operation Oct 30, 2024

adding operation generation

d7e2964

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

JaydipGabani force-pushed the operation-generate branch from 2947f62 to d7e2964 Compare November 5, 2024 00:13

fixing tests

f411663

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe reviewed Nov 5, 2024

View reviewed changes

JaydipGabani added 2 commits November 5, 2024 05:26

refactoring generate vap call

66626cf

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

calling generateVAPB after constraint is enforced

a7ae597

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

JaydipGabani requested a review from maxsmythe November 5, 2024 17:51

maxsmythe reviewed Nov 6, 2024

View reviewed changes

ritazh reviewed Nov 6, 2024

View reviewed changes

not updating annotation on CRD update

45db3f4

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

JaydipGabani force-pushed the operation-generate branch from 6e863dd to 45db3f4 Compare November 6, 2024 16:09

JaydipGabani requested review from maxsmythe and ritazh November 6, 2024 18:06

adding todo to follow up with enforcement point status

7c9881b

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

JaydipGabani mentioned this pull request Nov 6, 2024

Add non-clock-dependent annotation for VAPB generation #3683

Closed

ritazh approved these changes Nov 6, 2024

View reviewed changes

ritazh changed the title ~~feat: adding generate operation~~ feat: add generate operation and wait for VAPB generation Nov 7, 2024

maxsmythe approved these changes Nov 7, 2024

View reviewed changes

updating variable name to noDelay, updating decription to why we are …

a6ac8fd

…setting annotation on CT Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe approved these changes Nov 7, 2024

View reviewed changes

ritazh approved these changes Nov 7, 2024

View reviewed changes

JaydipGabani merged commit 1834741 into open-policy-agent:master Nov 7, 2024

JaydipGabani deleted the operation-generate branch November 7, 2024 23:08

BrewTestBot mentioned this pull request Dec 12, 2024

gator 3.18.0 Homebrew/homebrew-core#200877

Merged

sultetveny mentioned this pull request Feb 19, 2025

From v3.18.x CRD created by gatekeeper-audit with parameter --operation=generate (not by gatekeeper-controller-manager) #3827

Closed

	\| defaultWaitForVAPBGeneration \| (alpha) Wait to generate ValidatingAdmissionPolicyBinding after the constraint CRD is created. \| `30` \|
	\| defaultWaitForVAPBGeneration \| (alpha) Wait time in seconds before generating a ValidatingAdmissionPolicyBinding after a constraint CRD is created. \| `30` \|

Conversation

JaydipGabani commented Oct 5, 2024 • edited by ritazh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

maxsmythe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaydipGabani Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

JaydipGabani commented Oct 5, 2024 •

edited by ritazh

Loading

codecov-commenter commented Oct 5, 2024 •

edited

Loading

JaydipGabani Oct 30, 2024 •

edited

Loading

JaydipGabani Nov 5, 2024 •

edited

Loading