fix: enable cert rotation for audit by default

[JaydipGabani]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #2516

Special notes for your reviewer:

[JaydipGabani]

@maxsmythe this issue can also be resolve with removing --disable-cert-rotation from audit as you mentioned here. The reason I am disabling external-data for audit is that I am not sure what will happen in an upgrade if --disable-cert-rotation is set there and after if we remove it. Let me know, I can change it to remove that from the audit if there is no impact.

Additionally, I raised this PR against master thinking we will cherry-pick in respective releases from the master. Let me know if that is not the case. I will close this PR and open a new one against 3.11 release in that case.

cc: @sozercan

[codecov-commenter]

Codecov Report

Patch coverage has no change and project coverage change: -0.04% ⚠️

Comparison is base (ea842cd) 53.13% compared to head (fb17509) 53.10%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2875      +/-   ##
==========================================
- Coverage   53.13%   53.10%   -0.04%     
==========================================
  Files         135      135              
  Lines       11790    11790              
==========================================
- Hits         6265     6261       -4     
- Misses       5041     5044       +3     
- Partials      484      485       +1     

Flag	Coverage Δ
unittests	53.10% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maxsmythe]

This would be a breaking change for users.

Why not add the flag to audit by-default?

Can we only disable cert rotation if audit has external data enabled? I'm guessing via an if statement in the Helm chart?

[JaydipGabani]

Why not add the flag to audit by-default?

what do you mean?

Can we only disable cert rotation if audit has external data enabled? I'm guessing via an if statement in the Helm chart?

When we upgrade to 3.11 external-data is enabled by default, which would mean setting disable-cert-rotation to false. We can do this with if in helm chart. Is this the correct solution?

[JaydipGabani]

@maxsmythe @sozercan changed it to not disable cert rotation for audit by default.

[sozercan]

does this have any side effects since we'll be generating certificates twice (in audit and in controller-manager)?

[JaydipGabani]

does this have any side effects since we'll be generating certificates twice (in audit and in controller-manager)?

the only one I could think of was if there could be a race condition between audit and controller pod to write certs to secret. But I didn't notice anything concerning when I tested the change.

[maxsmythe]

does this have any side effects since we'll be generating certificates twice (in audit and in controller-manager)?

Actually we'd be generating certs 4 times (3 controller-manager pods, 1 audit pod). Worst case side effect should be an extra write to the API server for each pod, all but the first write should fail due to optimistic concurrency check failure (stale resourceVersion).

[ritazh]

Is #2121 (comment) no longer an issue?

I tried enabling cert-controller for both audit and webhook deployment, and I encountered weird issues

[maxsmythe]

Couldn't say. "weird issues" is too vague to know what was noticed, if it can be reproduced, or whether it's gone. I'd suggest trying it out and seeing if we notice anything "weird". There's nothing inherent in the concept that should be broken.

[JaydipGabani]

I can try running this with external data enabled to see if I face any errors.

[JaydipGabani]

Tested it! I didn't notice anything weird or face issues in running the audit or webhook.

@maxsmythe @ritazh

[sozercan]

LGTM

[maxsmythe]

LGTM